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PREFACE 

This  monograph  is  designed  to  summarize  and  synthesize  the  results 
of  a  series  of  studies  conducted  to  investigate  the  accuracy  of  savings 
statistics  as  obtained  by  consumer  survey  techniques.  It  is  based  on  re- 
search conducted  by  the  Consumer  Savings  Project  of  the  Inter-University 
Committee  for  Research  on  Consumer  Behavior,  a  project  financed  by  the 
Ford  Foundation  with  supplementary  assistance  from  the  United  States 
Department  of  Agriculture,  the  United  States  Department  of  Labor,  and 
later  from  the  National  Science  Foundation.  The  objectives  of  this  project 
were  threefold,  as  follows : 

(1)  To  determine  the  reliability  with  which  quantitative  information 
on  various  kinds  of  assets  and  debts  and  on  the  size  and  composition  of 
saving  and  dissaving  can  be  collected  from  consumers  by  survey  techniques. 

(2)  To  develop  procedures  for  obtaining  these  data  on  a  current  and 
continuing  basis  with  a  known  degree  of  reliability. 

(3)  To  begin  to  collect  consumer  financial  data  of  a  kind  not  hitherto 
available  and  of  strategic  importance  in  studying  the  decisions  consumers 
make  about  their  assets  and  debts,  spending  and  saving. 

The  first  of  these  objectives  is  the  concern  of  the  present  monograph, 
which  reviews  the  findings  of  the  various  subprojects  relating  to  the  relia- 
bility with  which  different  assets  and  debts  were  reported  in  the  different 
surveys.  In  addition,  an  attempt  is  made  to  incorporate  the  relatively 
small  amount  of  work  done  previously  on  this  subject,  to  provide  in  a 
single  source  a  relatively  comprehensive  review  of  present  knowledge  of 
the  reliability  of  consumer  reports  of  financial  assets  and  debts.  The  prin- 
cipal exception  is  the  1963  reliability  study  carried  out  in  conjunction  with 
that  year's  Federal  Reserve  Survey  of  Family  Financial  Characteristics, 
which  is  covered  only  briefly  here  because  the  results  of  that  study  are  not 
complete  at  the  time  of  this  writing. 

This  monograph  is  divided  into  eight  chapters.  A  general  discussion 
of  the  reliability  problem  as  applied  to  consumer  financial  surveys  is  pro- 
vided in  Chapter  I,  followed  in  Chapter  II  by  a  review  of  aggregative 
comparisons  of  data  collected  by  such  surveys.    Findings  relating  to  the 


reliability  of  individual  reports  of  assets  and  debts,  based  largely  on  the 
studies  of  the  Consumer  Savings  Project,  are  presented  in  Chapters  III-VI. 
The  reliability  of  reports  of  debt  is  discussed  in  Chapter  III,  that  of  time 
deposits  in  Chapter  IV,  and  that  of  life  insurance,  demand  deposits,  and 
farm  assets  in  Chapter  V.  The  effects  of  a  panel  operation  on  the  relia- 
bility of  such  reports  are  investigated  in  Chapter  VI,  based  on  two  such 
operations  which  were  conducted  as  an  extension  of  studies  covered  in 
earlier  chapters. 

Means  of  improving  the  reliability  of  consumer  survey  data  by  straight- 
forward survey  techniques  are  evaluated  in  Chapter  VII.  The  results  of 
various  experiments  designed  to  test  the  efficiency  of  such  approaches  are 
also  presented  in  that  chapter.  The  final  chapter  contains  a  summary 
discussion  of  the  significance  of  the  results  and  of  the  possible  directions 
for  future  work  on  survey  reliability  in  the  light  of  these  findings. 

This  monograph  does  not  purport  to  present  the  complete  details  and 
results  of  the  Consumer  Savings  Project,  not  even  in  the  methodological 
area.  This  is  hardly  necessary  since  five  monographs  on  the  project,  all 
part  of  the  series  on  Studies  in  Consumer  Savings  published  by  the  Bureau 
of  Economic  and  Business  Research  of  the  University  of  Illinois,  have 
preceded  this  volume.  Also,  one  other  study  is  still  in  progress. 

The  members  of  the  Inter-University  Committee  for  Research  on 
Consumer  Behavior  are: 

Lincoln  Clark,  New  York  University,  Secretary-Treasurer 
Robert  Ferber,  University  of  Illinois 
George  Katona,  University  of  Michigan 
Theodore  Newcomb,  University  of  Michigan 
Howard  Raiffa,  Harvard  University 
James  Tobin,  Yale  University 
Guy  Orcutt,  University  of  Wisconsin,  Chairman 
Raymond  Goldsmith  was  a  member  of  the  committee  until  he  left  in 
June,  1963,  for  an  OECD  assignment  in  Paris. 

The  monographs  in  this  series  are  research  reports.  The  Inter- 
University  Committee,  as  sponsor  of  this  research,  makes  every  effort  to 
ensure  both  the  quality  of  the  reports  and  their  orientation  toward  meet- 
ing a  real  need.  Nevertheless,  the  findings  reported  in  this  way  summarize 
conclusions  arrived  at  by  project  staff  and  do  not  necessarily  represent  the 
individual  or  collective  views  of  the  members  of  the  Inter-University 
Committee. 

Guy  Orcutt,  Chairman 
Inter-University  Committee 
for  Research  on  Consumer  Behavior 
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I.   THE  RELIABILITY  PROBLEM  IN  CONSUMER  SURVEYS 


The  principal  focus  of  this  chapter  is  on  the  nature  and  measurement 
of  survey  errors  in  consumer  financial  studies.  As  a  basis  for  such  a  dis- 
cussion, however,  it  would  seem  desirable  to  define  relevant  terms  and  to 
enumerate  some  of  the  principal  conceptual  problems  involved  in  the 
measurement  of  savings.  Accordingly,  the  first  part  of  this  chapter  will  be 
concerned  with  the  general  question  of  the  measurement  of  consumer 
savings  and  saving,  and  the  second  part  will  be  devoted  to  problems  aris- 
ing in  the  measurement  process,  specifically  that  of  data  reliability. 

Measurement  Problems 
Savings  and  Saving 

In  any  study  of  savings,  or  of  saving,  a  clear  distinction  has  to  be  made 
between  these  two  terms.  By  savings  is  meant  the  stock  of  assets  and  debts 
at  a  particular  moment  in  time.  Saving  refers  to  the  accumulation  of 
assets  and  debts  over  a  period  of  time.  In  other  words,  saving  is  a  flow 
concept,  representing  the  change  in  the  stock  of  assets  and  debts  between 
two  points  in  time.1 

Both  saving  and  savings  can  be  positive  or  negative.  Saving  is  nega- 
tive if  expenditures  plus  debt  exceed  income  plus  other  receipts  during 
the  given  period;  in  such  a  case,  there  is  dissaving.  In  a  similar  fashion, 
savings  are  negative  if  the  stock  of  debts  exceeds  the  stock  of  assets  at  the 
given  moment. 

Alternative  Definitions  of  Savings 

Definitions  of  savings,  and  of  saving,  vary  in  scope  and  complexity. 
No  single  definition  is  adequate  for  all  purposes.  Data  corresponding  to 
the  more  general  definitions  tend  both  to  be  more  difficult  to  obtain  and 
to  involve  more  frequent  and  more  doubtful  assumptions. 

1  To  be  sure,  savings  could  also  represent  the  plural  of  saving.  However,  for 
the  sake  of  clarity  this  use  of  "savings"  will  be  avoided,  invoking  —  if  necessary  — 
poetic  license  to  do  so. 
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First,  and  simplest,  is  a  definition  that  equates  savings  with  liquid 
assets.  Liquid  assets  include  currency,  checking  accounts,  United  States 
savings  bonds,  and  savings  accounts  in  their  various  forms  (including 
certificates  of  deposit  and  share  and  investment  accounts  in  commercial 
and  mutual  banks,  savings  or  building  and  loan  associations,  postal  sav- 
ings, and  credit  unions) .  All  of  these  assets  have  the  following  two  prop- 
erties: they  either  consist  of  cash  or  are  immediately  convertible  into 
cash,  and  they  are  in  fixed  dollars.  The  latter  means  that  the  number  of 
dollars  obtainable  from  an  account  equals  the  sum  of  the  net  deposits  plus 
interest,  if  any.  The  amount  does  not  fluctuate  with  changes  in  prices  or 
market  or  business  conditions. 

A  more  inclusive  definition  of  savings  is  in  terms  of  financial,  or  in- 
tangible, assets.  The  alternate  term,  "intangible,"  is  the  more  descriptive 
because  it  includes,  in  effect,  assets  with  no  tangible  value  of  their  own. 
Included  are  not  only  liquid  assets  but  all  other  financial  assets  as  well  — 
marketable  government  and  other  bonds,  common  and  preferred  stocks 
(such  as  insurance  company  stocks),  annuities,  mortgages  and  debt  (on 
both  the  debit  and  the  credit  sides) ,  and  pensions. 

More  inclusive  still  is  to  incorporate  houses,  cars,  and  other  durable 
goods,  the  latter  valued  generally  at  market  (resale)  prices.  There  are 
two  main  reasons  for  including  such  expenditures  in  a  savings  definition. 
One  is  that  to  many  people  these  purchases  are  not  really  expenditures. 
Especially  when  prices  are  rising  appreciably,  or  are  expected  to  do  so, 
the  purchase  of  durable  goods  may  represent  a  substitute  form  of  savings.2 
If  prices  rise  sufficiently,  ownership  of  durables  can  indeed  serve  to  main- 
tain one's  capital  or  even  to  bring  about  substantial  capital  appreciation, 
as  was  the  case  during  and  shortly  after  World  War  II. 

A  second  reason  for  treating  durables  as  a  form  of  savings  is  that  such 
goods  almost  always  possess  resale  value,  which  for  houses  and  cars  may 
well  represent  the  major  part  of  a  consumer's  capital.  That  such  a  tend- 
ency actually  exists  on  the  part  of  consumers  to  treat  investments  and 
purchase  of  durables  as  substitute  forms  of  savings  is  supported  by  studies 
which  show  that  the  sum  of  the  two  bears  a  more  stable  relationship  to 
income  than  does  each  component  separately.3  In  addition,  one  could 
cite  socio-psychological  studies  which  indicate  that  ownership  of  many 
durables  possess  status  and  prestige  as  symbolic  of  wealth  as  ownership  of 

2  Irwin  Friend  and  Robert  Jones,  "The  Concept  of  Saving,"  in  Proceedings 
of  the  Conference  on  Consumption  and  Saving,  Vol.  2  (Philadelphia:  University 
of  Pennsylvania  Press,  1960),  pp.  336-59. 

8  E.  S.  Maynes,  "The  Relationship  Between  Tangible  Investment  and  Consumer 
Saving,"  Review  of  Economics  and  Statistics,  Vol.  41,  No.  3  (August,  1959), 
pp.  278-93. 
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financial  assets.4  Analytically,  too,  such  a  definition  has  proved  useful, 
leading  to  provocative  theories  of  consumer  behavior.5 

Conceptually,  however,  a  slight  extension  of  the  previous  definition  of 
savings  would  seem  desirable  because  of  the  fact  that  certain  so-called 
nondurables,  such  as  suits  and  overcoats,  also  possess  resale  value  and  may 
have  a  longer  life  span  than  many  items  classified  as  durables.  Such  an 
extension  leads  to  the  most  inclusive  definition  of  savings,  which  may  be 
termed  the  net-worth  concept.  According  to  this  definition,  savings  would 
include  the  resale  value  of  all  tangible  goods  in  addition  to  financial  hold- 
ings, deducting  debt  and  other  financial  obligations.  In  other  words,  this 
definition  covers  the  material  value  of  one's  entire  possessions. 

Although  the  simplest  of  all  theoretically,  this  definition  is  rarely,  if 
ever,  used  in  empirical  work  because  of  the  difficult  valuation  problems 
that  arise.  Perplexing  as  it  is  to  value  items  for  which  relatively  organized 
markets  exist  (such  as  houses  and  cars) ,  valuing  items  such  as  clothing  and 
household  furnishings,  which  come  in  innumerable  sizes,  styles,  and 
shapes,  becomes  a  virtually  hopeless  task.  In  the  present  volume,  no 
attention  is  given  to  the  valuation  of  consumer  goods,  durable  or  other- 
wise, because  the  main  focus  is  on  the  collection  of  financial  assets  data. 

Estimating  Saving 

By  virtue  of  the  two  different  ways  of  defining  saving,  a  survey  pro- 
cedure designed  to  estimate  saving  —  whether  by  consumer  survey  or  by 
institutional  canvass  —  can  also  follow  either  of  two  approaches,  the 
specification  of  which  has  major  ramifications  for  numerous  aspects  of  the 
survey  procedure. 

The  first  approach,  the  more  direct  of  the  two,  entails  asking  the 
respondent  (or  the  institution)  to  report  directly  the  amount  of  saving (s) 
during  a  period.  Thus,  by  this  approach  the  change  in  one's  checking 
account  (s)  during  the  first  quarter  of  the  year  would  be  ascertained  in  a 
consumer  survey  by  asking  the  sample  member  a  question  such  as:  "How 
much  change  was  there  in  the  balance  in  your  checking  account  (s)  be- 
tween January  1  and  April  1?"  This  approach,  which  provides  a  direct 
estimate  of  saving,  we  shall  call  the  change  approach. 

The  second  approach  is  to  estimate  saving  as  the  computed  difference 
between  holdings  of  particular  assets  or  debts  reported  at  the  beginning 
(H0)  and  at  the  end  (Hi)  of  a  period. 

4  Thus,  in  many  neighborhoods,  especially  suburbs,  ownership  of  a  high-priced 
car  was  for  a  long  time  a  far  more  important  symbol  of  success  and  affluence  than 
ownership  of  savings  accounts  or  stocks  —  which  could  hardly  be  parked  outside 
one's  house  for  neighbors  to  envy. 

5  For  example,  Milton  Friedman,  A  Theory  of  the  Consumption  Function 
(Princeton:   Princeton  University  Press,  1956). 


4         The  Reliability  of  Consumer  Reports  of  Financial  Assets  and  Debts 

By  this  approach,  the  holdings  approach,  each  estimate  requires  an- 
swers to  two  questions : 

H0'      "What  was  the  balance  in  your  checking 

account(s)  on  January  1?" 
Hx :      "What  was  the  balance  in  your  checking 

account  (s)  on  April  1?" 

These  two  questions  could  be  asked  on  a  single  interview  (on  or  after 
April  1 ) ,  or  each  could  be  asked  in  a  separate  interview,  the  first  on  or 
shortly  after  January  1  and  the  second  on  or  shortly  after  April  1.  To 
obtain  this  information  in  the  course  of  a  single  interview  is  clearly  most 
economical.  Unfortunately,  this  economy  comes  only  at  a  price,  since 
such  factors  as  memory  errors  and  lack  of  information  tend  to  reduce  the 
reliability  of  the  data  furnished  for  the  earlier  point  in  time.  Evidence  to 
this  effect  will  be  provided  in  later  chapters. 

Apart  from  this  question,  the  selection  of  one  of  these  two  general 
approaches  in  a  particular  case  is  a  difficult  matter.  From  a  purely  survey 
point  of  view,  the  change  approach  is  the  easier.  It  requires  asking  only 
one  question  —  a  question  which  is  likely  to  be  less  sensitive  than  one  on 
holdings;  and  it  shifts  the  burden  of  computation  from  the  survey  staff  to 
the  respondent.  Further  thought,  however,  indicates  that  these  advan- 
tages in  favor  of  the  change  approach  are  by  no  means  clear-cut,  and 
they  may  not  even  be  advantages.  Thus,  the  question  put  to  the  respon- 
dent is  not  really  one  but  is  in  effect  three,  with  the  very  rare  exception 
occurring  when  the  respondent  has  actually  computed  his  saving  for  the 
period  in  question.  The  questions  are : 

( 1 )  "Look  up  (or  recall)  the  balance  in  your  checkbook  on  January  1." 

(2)  "Look  up  (or  recall)  the  balance  in  your  checkbook  on  April  1." 

(3)  "Compute  the  difference  and  tell  me  what  it  is." 

The  first  of  these  questions  implicitly  places  strain  on  the  respondent's 
willingness  to  cooperate  because  he  is  being  asked  to  recall  a  figure  at 
least  as  old  as  the  period  under  study  —  in  this  case  three  months.  The 
first  two  questions  taken  together  pose  another  problem,  one  particularly 
pertinent  to  reliability:  has  the  respondent  selected  comparable  balances? 
Thus,  if  the  balance  at  one  date  includes  a  mortgage  payment  for  that 
month  and  the  balance  for  the  other  date  selected  is  exclusive  of  the 
mortgage  payment  for  that  month,  the  reported  saving  will  be  biased.  Of 
course,  a  similar  danger  arises  with  the  holdings  approach  except  that,  as 
will  be  noted  later,  the  danger  is  not  so  great,  apparently  because  the 
explicit  nature  of  the  question  helps  both  the  interviewer  and  the  respon- 
dent to  focus  on  the  matter  of  comparability. 

The  third  of  these  questions  also  contains  a  hidden  danger,  namely, 
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the  assumption  that  the  respondent  can  subtract  correctly.6  The  condi- 
tions under  which  these  subtractions  take  place  are  not  usually  conducive 
to  a  high  degree  of  accuracy,  being  made  mentally  and  generally  very 
quickly.   Errors  in  arithmetic  are  therefore  not  unlikely. 

From  these  various  points  of  view,  the  holdings  approach  would  ap- 
pear to  offer  an  advantage.  In  addition,  the  holdings  approach  clearly 
provides  some  extra  highly  pertinent  data,  namely,  savings  in  these  par- 
ticular holdings,  thereby  enabling  saving  to  be  related  to  savings.  It  is  pre- 
cisely this,  however,  which  tends  to  be  the  principal  disadvantage  of  the 
holdings  approach.  The  request  for  figures  on  holdings  is  one  of  the  most 
personal  questions  that  can  be  asked.  If  there  is  any  doubt  in  the  re- 
spondent's mind  regarding  the  value  or  authenticity  of  the  study  or  of  the 
confidentiality  of  the  data,  the  result  is  likely  to  be  a  refusal  or  possibly 
the  giving  of  erroneous  information. 

For  the  sake  of  completeness,  a  third  method  of  estimating  savings 
should  be  mentioned,  although  it  is  not  generally  reliable  with  consumer 
surveys.  This  method,  which  we  may  call  the  income-expenditure  ap- 
proach, seeks  to  estimate  saving  by  ascertaining  total  income  and  deduct- 
ing from  it  expenditures  for  current  consumption,  and  in  some  cases 
expenditures  for  durable  goods  as  well.  The  method  is  used  by  the  United 
States  Department  of  Commerce  in  estimating  aggregate  personal  savings. 
In  that  particular  case,  estimates  of  aggregate  income  and  of  aggregate 
consumption  expenditures  are  obtained,  with  aggregate  saving  derived 
as  the  residual. 

When  applied  to  consumer  surveys,  the  method  is  subject  to  consider- 
able error  because  saving  generally  constitutes  such  a  small  proportion  of 
total  income.  As  a  result,  any  error  in  the  estimates  of  income  or  of  ex- 
penditures tends  to  be  magnified  many  times  in  the  residual  estimate  of 
saving.  The  Department  of  Commerce  estimate  of  saving  is  subject  to 
much  the  same  difficulty,  except  that  it  might  be  argued  that  for  a  single 
aggregate  estimate  the  possibility  of  large  errors  is  low,  due  in  part  to  the 
availability  of  crosschecks  in  the  national  income  accounts. 

Institutions  Versus  Consumers  as  the  Source  of  Data 

With  the  exception  of  currency,  all  forms  of  assets  and  debts  involve 
some  organization  or  individual  other  than  the  consumer  unit,  generally 
an  institution  or  a  government.  Since  saving  in  the  form  of  currency  is 
negligible  from  year  to  year,  two  alternative  sources  of  information  on 

6  It  also  involves  the  assumption  that  positive  and  negative  changes  can  be 
distinguished  from  each  other,  an  error  that  seems  to  be  made  infrequently  but  is 
of  substantial  proportions  when  it  does  occur. 
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consumer  saving  would  appear  to  exist  —  the  consumer  himself  or  the 
appropriate  financial  institution.7 

In  theory,  either  source  could  supply  the  data  needed  to  estimate 
aggregate  consumer  savings,  or  saving;  at  least,  the  data  are  there.  Sim- 
ilarly, data  could  be  obtained  from  either  source  on  the  size  distribution 
of  individual  holdings,  although  not  of  all  holdings  of  a  particular  asset 
or  debt.  Thus,  a  random  selection  of  savings  accounts  of  individuals  in 
banks  and  savings  institutions  would  yield  an  unbiased  estimate  of  the 
distribution  of  individual  accounts  as  ascertained  from  a  consumer  survey, 
abstracting  from  survey  errors.  However,  such  a  random  selection  would 
not  yield  an  unbiased  estimate  of  the  size  distribution  of  total  savings  of 
consumer  units  in  the  form  of  savings  accounts. 

For  the  latter  purpose,  as  well  as  for  any  data  involving  the  relation- 
ship between  ownerships  of  different  types  of  savings  (or  saving),  the 
consumer  survey  is  the  only  source.  A  consumer  survey  is  also  required 
if  saving  is  to  be  related  to  consumer  characteristics,  attitudes,  or  plans, 
except  for  those  few  instances  in  which  these  characteristics  have  been 
recorded  by  the  institution.8  In  addition,  there  is  one  aggregate  for  which 
institutional  data  are  clearly  inadequate  and  which  is  only  obtainable  by 
consumer  survey,  namely,  saving  in  farm  and  nonfarm  unincorporated 
business. 

On  the  other  hand,  certain  data,  pertaining  primarily  to  institutional 
characteristics  of  consumer  savings,  can  only  be  obtained  from  institutions. 
These  would  include  such  data  as  the  breakdown  of  demand  deposits 
among  individuals,  nonprofit  organizations,  businesses,  and  so  forth,  and 
the  distribution  of  ownership  of  mortgages  among  consumers  and  other 
holders.  Also  included  would  be  data  which,  although  theoretically  avail- 
able from  consumers,  are  not  likely  to  be  reported  with  much  reliability, 
such  as  accumulated  saving  in  the  form  of  life  insurance  and  pension 
plans. 

For  many  purposes,  therefore,  these  two  data-collection  sources  are 
essentially  complementary.  In  actual  practice,  however,  situations  fre- 
quently arise  in  which  the  data  requirements  could  be  met  by  going  to 
either  source.  In  addition,  questions  have  been  raised  regarding  the  re- 
liability of  financial  data  obtained  in  consumer  surveys  (the  subject  of  the 
present  work).  As  a  result,  considerable  controversy  exists  regarding  the 
pros  and  cons  of  approaching  each  source. 

7  For  the  sake  of  simplicity,  we  shall  assume  for  the  time  being  that  consumer 
assets  and  debts  are  held  only  by  institutions,  including  governments,  and  not  by 
other  consumers.  With  the  possible  exception  of  personal  loans,  this  assumption  is 
no  doubt  true. 

8  Even  then  a  consumer  survey  may  be  required,  since  such  data  are  generally 
of  marginal  value  to  the  institution's  operations  and  little  effort  is  made  to  collect 
them  accurately  or  to  keep  them  up-to-date. 
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Two  basic  arguments  are  advanced  in  favor  of  contacting  institutions 
rather  than  consumers  for  financial  data:  they  are  fewer,  and  they  are 
accustomed  to  supplying  financial  information.  For  any  single  type  of 
savings,  it  is  clearly  more  efficient  to  sample  the  records  of  a  relatively 
few  institutions  than  to  contact  hundreds  or  thousands  of  consumers  for 
comparable  information.  In  such  a  case,  contacting  institutions  is  likely 
to  be  quicker  as  well  as  much  cheaper.  Furthermore,  institutions  are 
likely  to  be  more  receptive  to  such  requests  —  since  they  prepare  and 
furnish  data  on  their  other  operations  frequently  - —  and  are  also  likely 
to  provide  such  data  more  accurately  because  of  the  records  they  keep. 
Hence,  to  the  extent  that  data  can  be  obtained  from  institutions,  this 
source  should  be  preferred. 

Yet  these  arguments  do  not  detract  from  the  indispensability  of  con- 
sumer interviews.  Interest  in  the  distribution  of  single  accounts  or  more 
generally  in  the  distribution  of  single  holdings  of  a  single  asset  —  without 
reference  to  consumer  characteristics,  ownership  of  holdings  of  the  same 
asset,  or  holdings  of  different  assets  —  represents  a  small  minority  of  con- 
sumer financial  studies.  Even  in  this  minority  of  situations,  certain  hold- 
ings (such  as  equity  in  an  unincorporated  business)  can  only  be  obtained 
by  consumer  surveys.  When  attention  is  broadened  to  include  several 
assets  and  debts  or  all  holdings  of  a  single  asset  (such  as  a  consumer  unit's 
total  investment  in  government  savings  bonds),  consumer  surveys  are 
generally  the  only  answer.  Although  they  are  obtainable  theoretically 
from  institutions  also,  the  multiplicity  of  institutions  and  of  the  record- 
keeping practices  of  different  financial  intermediaries,  combined  with  the 
absence  of  attempts  to  keep  names  and  addresses  up-to-date,  makes  the 
collating  problem  insoluble  for  all  practical  purposes. 

Consumer  surveys  would  also  seem  superior  if  interest  is  in  obtaining 
only  univariate  size  distributions  of  different  assets  and  debts  separately, 
e.g.,  a  size  distribution  of  checking  accounts  and  a  size  distribution  of 
savings  accounts  and  a  size  distribution  of  government  savings  bonds,  and 
so  forth.  This  information  could  be  obtained  by  sampling  the  records  of 
institutions  in  a  corresponding  number  of  different  financial  areas.  How- 
ever, since  information  on  all  holdings  could  be  also  obtained  in  a  single 
visit  to  a  consumer  unit,  abstracting  from  the  data  reliability  problem, 
the  consumer  survey  is  likely  to  be  more  economical  in  such  a  case  and 
perhaps  quicker,  depending  on  the  number  of  holdings  involved. 

The  two  basic  problems  associated  with  the  use  of  consumer  surveys 
are  data  reliability  and  cost.  The  data  reliability  problem  stems  from  a 
number  of  factors  which  lead  some  people  either  not  to  provide  any 
information  or  to  provide  inaccurate  information.  Since  these  factors  will 
be  considered  in  later  chapters,  it  need  only  be  said  here  that  their  net 
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result  is  to  produce  estimates  of  financial  holdings  biased  not  only  in  the 
aggregates  but  in  the  distribution  of  these  holdings  as  well.  These  biases 
are  substantial  enough  to  make  questionable  the  advisability  of  devoting 
extensive  resources  and  analytical  talent  to  the  collection  of  such  data,  if 
the  biases  are  not  detected  and  corrected. 

Cost  also  is  a  major  problem,  partly  because  of  the  difficulty  of  con- 
tacting all  sample  members  and  partly  because  of  the  inherently  high  ex- 
penses of  the  field  interviews.  The  confidential  nature  of  the  data  re- 
quested precludes  anything  but  personal  contact,  at  least  on  an  initial 
interview.  The  interview  itself  requires  a  personable  and  high-caliber 
individual  as  well  as  one  with  considerable  skill  in  asking  questions  and 
recording  the  replies.  The  result  costwise  is  that  field  charges  alone  can 
average  anywhere  between  $15  and  $25  per  interview,  and  the  overall 
cost  of  a  nationwide  survey  of  about  3,000  consumers  can  aggregate  be- 
tween $100,000  and  $200,000. 

Despite  these  limitations,  the  fact  remains  that  consumer  surveys  are 
indispensable.  With  increasing  attention  being  devoted  to  problems  of 
consumer  behavior,  and  particularly  to  consumer  financial  behavior,  such 
surveys  are  likely  to  be  required  even  more  in  the  future.  The  basic  prob- 
lems therefore  become  how  to  use  such  surveys  most  effectively  and  how 
to  overcome  their  limitations.  In  the  past,  attention  has  focused  primarily 
on  the  improvement  of  survey  procedures  —  better  interviewers,  more  in- 
tensive training  sessions,  simpler  questionnaires,  special  attempts  to  inter- 
view initial  refusals  and  noncontacts,  and  so  on.  Improvement  along  these 
lines  will  no  doubt  be  helpful  and  may  produce  significantly  more  reliable 
data.  At  the  same  time,  a  broader  perspective  may  well  be  worth  con- 
sidering. Such  a  perspective  was  provided  by  Irwin  Friend's  proposal  in 
the  late  1950's  designed  to  take  advantage  of  some  of  the  best  features  of 
contacting  both  sources.9  The  basis  of  this  proposal  is  sampling  institu- 
tional records  for  data  on  consumer  holdings  and  then  contacting  the  con- 
sumers sampled  for  such  classifying  information  as  age,  occupation,  family 
size,  and  so  forth.  To  quote  Friend : 

The  general  plan  to  be  followed  for  each  institutional  body  of  data  to  be  cov- 
ered is,  first,  to  select  a  sample  of  institutions  and,  second,  to  select  a  sample  of 
accounts  within  each  institution,  transcribing  names  and  addresses  of  the  owners 
and  dollar  amounts  of  the  accounts  at  the  beginning  and  end  of  the  year.  Though 
a  number  of  variations  in  procedure  are  possible  and  should  be  tested,  the  final 
step  would  probably  involve  contacting  the  individuals  directly  to  obtain  occupa- 

9  Irwin  Friend,  "Institutional  Data  as  a  Source  of  New  Information  for  Use  in 
Social  Accounting  Systems,"  paper  delivered  at  the  Conference  on  Research  in 
Income  and  Wealth,  November  6-7,  1959,  and  printed  in  The  Flow-of-Funds  Ap- 
proach to  Social  Accounting,  Studies  in  Income  and  Wealth,  Vol.  26,  National 
Bureau  of  Economic  Research  (Princeton:  Princeton  University  Press,  1962). 
Friend  had  been  working  on  this  idea  for  some  years  before  this  paper  was  given. 
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tion  (including  occupation  of  the  head  of  the  family  if  different  from  the  indi- 
vidual), income  class,  and  other  significant  family  characteristics.  Such  contacts 
can  be  effected  by  mail,  with  telephone  and  interview  follow-ups  if  necessary.10 

In  that  paper,  Friend  describes  one  such  case  study  in  which  stock- 
holder characteristics  were  ascertained  by  a  mail  questionnaire,  yielding 
almost  a  two-thirds  response  without  any  follow-up. 

This  method  has  the  advantage  of  yielding  highly  accurate  financial 
statistics,  which  are  then  related  to  selected  consumer  characteristics.  It 
is  also  likely  to  be  far  more  economical  than  a  personal  interview  survey. 
The  method  is  effective  in  allocating  single  holdings  of  assets  and  debts 
among  consumer  units.  The  emphasis  has  to  be  on  "single,"  however, 
because  it  is  not  feasible  to  match  different  assets  via  institutional  records 
to  obtain  a  consumer  unit's  total  portfolio,  or  even  to  match  different 
holdings  of  the  same  asset  to  derive  a  consumer  unit's  total  holdings  of 
that  asset. 

Apart  from  this,  the  method  possesses  other  disadvantages,  some  not 
apparent  offhand.  For  one  thing,  it  does  not  do  away  with  a  consumer 
survey,  and  hence  it  leaves  survey  problems  and  biases  still  to  be  con- 
sidered.   In  this  respect,  the  following  assertion  is  open  to  question: 

.  .  .  certain  basic  family  characteristics  such  as  occupation  can  be  obtained  quite 
reliably  from  consumer  surveys  and  even  income,  which  is  more  difficult  to  mea- 
sure, has  a  much  smaller  margin  of  error  attached  to  it  than  data  on  saving.11 

Such  errors  obtained  in  the  few  studies  that  were  designed  for  this 
purpose  all  pertain  to  personal  interviews.  To  acquire  such  data  by  mail, 
or  by  telephone,  is  likely  to  involve  much  larger  errors  even  in  replies 
about  consumer  characteristics,  because  these  are  the  techniques  that  are 
most  conducive  to  misunderstanding,  replies  by  wrong  respondents,  dis- 
trust of  auspices,  and  so  on.  The  nature  and  magnitude  of  these  errors 
have  yet  to  be  investigated. 

Furthermore,  it  is  not  clear  that  information  about  a  characteristic 
such  as  occupation  can  be  secured  much  more  reliably  in  a  consumer  inter- 
view than  data  on  saving,  even  by  personal  interview,  because  of  inherent 
biases  and  prejudices  concerning  the  subject.  To  take  one  example,  very 
few  "janitors"  are  reported  on  consumer  surveys  —  they  are  all  "super- 
intendents of  operations,"  or  something  similar.  The  little  validation  work 
that  has  been  done  on  this  subject,  as  yet  unpublished,  points  to  a  strong 
upward  occupational  bias,  even  in  the  United  States  Census  data.  That 
this  bias  is  likely  to  be  still  larger  on  a  mail  questionnaire  is  a  strong 
probability. 

Finally,  it  might  be  noted  that  only  a  limited  amount  of  information 

10  Ibid.,  pp.  5-6. 

11  Ibid.,  p.  7. 
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can  be  requested  on  a  small  questionnaire  and  still  produce  high  response 
rates.  If  much  detail  or  probing  is  desired,  the  data  will  probably  have 
to  be  obtained  by  personal  interview.  Even  when  relatively  little  infor- 
mation is  required  from  consumer  units  and  response  rates  are  high,  a 
reliable  survey  will  undoubtedly  entail  personal  interview  follow-ups  with 
a  subsample  of  the  mail  nonrespondents. 

These  limitations  notwithstanding,  the  method  advocated  by  Friend 
has  served  to  broaden  the  horizons  of  possible  solutions  to  the  problems  of 
collecting  consumer  saving  data.  It  fulfills  a  real  need  by  furnishing  a 
much  more  reliable  method  than  formerly  existed  for  ascertaining  charac- 
teristics of  holders  of  particular  assets  or  debts  —  provided  that  possible 
biases  are  recognized  and  appropriate  precautionary  steps  are  taken. 

Survey  Errors 
Sampling  Versus  Nonsampling  Errors 

Estimates  derived  from  surveys  are  hardly  likely  to  coincide  with  the 
true  values  in  the  population,  for  two  reasons.  One  is  that  the  sampling 
process  produces  some  deviation  between  the  sample  estimate,  y  say,  and 
the  true  value,  x,  simply  because  the  sample  is  only  a  part  (and  generally 
a  very  small  part)  of  the  population.  These  so-called  sampling  errors  can 
be  measured  if  a  probability  sampling  procedure  is  used.  With  these 
sampling  designs,  the  probability  of  securing  estimates  differing  by  given 
amounts  from  the  true,  unknown  population  value  can  be  computed, 
given  certain  basic  information  such  as  the  sampling  technique,  sample 
size,  and  population  variances  (usually  estimates  of  these  variances) . 

The  derivation  of  sampling  error  formulas  has  become  essentially  an 
exercise  in  the  mathematics  of  probability,  although  in  some  cases  the 
mathematical  problems  can  be  intricate.  To  be  sure,  in  many  instances 
a  nonprobability  design  is  employed  in  a  survey,  a  design  for  which  the 
probabilities  of  selection  are  not  known.  In  those  instances,  the  sampling 
error  cannot  be  estimated  (although  the  usual  sample  precision  formulas 
are  often  applied  anyway) .  The  fact  remains,  however,  that  the  sampling 
error  problem -has  been  reduced  virtually  to  an  exact  science,  and  a  con- 
siderable literature  is  available  on  optimum  selection  of  sample  designs 
for  particular  purposes.  Practicable  theories  exist  for  measuring  and  con- 
trolling sampling  errors,12  so  that  in  a  particular  survey  the  precision  can 
be  set  at  any  desired  figure,  given  adequate  resources. 

The  second  reason  why  survey  estimates  deviate  from  the  true  popula- 
tion figure  is  the  myriad  of  other  types  of  errors  that  can  occur  in  such  an 

12  See,  for  example,  M.  H.  Hansen,  W.  N.  Hurwitz,  and  W.  G.  Madow, 
Sample  Survey  Methods  and  Theory,  Vols.  1  and  2  (New  York:  Wiley,  1953)  or 
W.  E.  Deming,  Sample  Design  for  Business  Research   (New  York:    Wiley,   1960). 
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operation  —  most  of  which  have  nothing  to  do  with  the  sampling  pro- 
cedure used  —  or  even  do  occur  whether  or  not  the  survey  is  based  on  a 
sample.  These  errors  include  such  varied  forms  as  recording  of  mistakes 
by  interviewers,  misinterpretation  of  replies,  coding  mistakes,  errors  in 
tabulation,  and  errors  in  computation.  There  are  so  many  varieties  that 
the  most  appropriate  designation  for  them  is  errors  other  than  sampling 
errors,  or  nonsampling  errors. 

Perhaps  the  one  general  characteristic  of  these  errors  is  that  from  an 
estimation  point  of  view  they  all  may  serve  to  bias  the  survey  estimates.  In 
the  absence  of  nonsampling  errors,  surveys  can  be  designed  to  yield 
unbiased  estimates  of  the  true  population  values,  meaning  that  as  the  size 
of  the  sample  approaches  that  of  the  population  the  survey  estimate  con- 
verges asymptotically  to  the  true  value.  When  nonsampling  errors  enter 
the  picture,  however,  this  is  no  longer  the  case,  for  these  errors  or  biases 
generally  have  little  to  do  with  sample  size.  As  we  shall  see  shortly,  the 
magnitude  of  some  of  these  biases  tends,  if  anything,  to  increase  with 
sample  size. 

Since  these  biases  are  usually  not  measurable  and  are  often  not  even 
suspected  to  exist,  the  sampling  property  of  unbiased  estimates  can  be- 
come artificial.  In  general,  the  larger  is  the  relative  importance  of  non- 
sampling  errors,  the  less  meaningful  becomes  the  sampling  property  of 
unbiasedness.  This  is  not  to  deny  that  this  property  has  practical  value, 
for  control  of  sampling  errors  is  desirable  in  any  event.13 

Effect  of  Bias  on  Survey  Estimates 

Biases  affect  survey  data  in  two  ways:  they  distort  the  population 
estimate  obtained  from  the  survey  data,  and  they  distort  the  reliability  of 
the  estimate.  In  the  first  instance,  the  expected  value  of  the  sample- 
based  estimate  will  be  biased,  even  if  the  estimate  otherwise  would  be 
unbiased.    In  other  words,  if  x  is  the  true  value  of  the  statistic  in  the 


13  A  distinction  should  be  made  between  biased  sampling  estimates  and  biases 
or  nonsampling  errors.  In  the  former  case,  reference  is  to  a  sampling  procedure, 
that  is  at  times  more  efficient,  of  using  an  estimation  procedure  which  yields  a 
biased  estimate  in  order  to  achieve  certain  other  properties,  such  as  reduced 
sampling  variance  relative  to  that  of  the  unbiased  estimating  procedure.  Such  is 
the  case,  for  example,  with  ratio  estimates,  in  which  instances  the  bias  is  incorpo- 
rated as  part  of  the  sampling  procedure;  its  magnitude  is  known  and  hence  can 
be  eliminated  in  the  final  analysis.  As  was  noted  previously,  biases  due  to  non- 
sampling  considerations  are  not  often  known  or  controllable,  and  they  are  rarely 
introduced  deliberately  — ■  certainly  not  to  improve  efficiency  of  estimation.  Since 
this  monograph  is  concerned  exclusively  with  nonsampling  errors,  possible  con- 
fusion arising  from  the  use  of  "bias"  in  two  different  senses  is  not  likely  to  arise. 
Henceforth,  "bias"  will  refer  to  the  nonsampling  errors  in  a  survey  operation.  (See, 
for  example,  Hansen,  Hurwitz,  and  Madow,  op.  cit.,  Vol.  1,  Chapters  IV  and 
VIII.) 
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population,  the  sample  estimate  of  x,  say  y,  will  deviate  from  the  true 
value,  on  the  average,  by  a  certain  non-zero  quantity,  e. 

Various  error  models  can  be  set  up  to  explore  the  effects  of  bias.  For 
the  present  purposes,  it  is  sufficient  to  base  such  a  model  on  the  following 
simple  relationship : 

yi  X{  ~~ l     €{) 

where  y\  is  the  observed  (or  reported)  value  for  the  zth  sample  member, 
Xi  is  the  true  value  of  this  observation,14  and  ei  represents  the  discrepancy 
in  the  report,  defined  as  yi  —  X{. 

Correspondingly,  we  may  define  the  means  of  these  variables,  which 
are  related  to  each  other  in  a  fashion  similar  to  those  of  the  individual 
observations,  namely: 

(1.2)  y  =  x  +  e. 

If  there  are  no  errors  in  the  individual  observations,  a  and  e  are  zero; 
e  may  also  be  zero  even  if  errors  are  present  in  the  individual  observations 
but  tend  to  offset  each  other.  Alternatively,  e  may  be  small  relative  to  x, 
in  which  case  bias  arising  from  the  use  of  y  as  an  estimate  of  x  may  go 
unnoticed.  However,  if  e  is  large  relative  to  x  the  bias  may  be  noticed 
even  in  the  absence  of  validation  information,  as  has  been  the  case  with 
many  past  saving  surveys. 

Biases  may  affect  the  reliability  of  an  estimate  as  well  as  the  estimate 
itself.  Bias  serves  to  increase  the  standard  error  of  estimate  of  a  particular 
statistic,  with  the  result  that  confidence  interval  estimates  turn  out  to  be 
faulty.  As  is  shown  in  various  sampling  texts,15  if  bias  is  present  in  the 
mean  (e  9^  0),  the  true  variance,  or  mean  square  error,  of  the  mean  is 
measured  in  terms  of  the  variance  of  the  observations,  yi,  about  the  true 
mean,  x,  i.e.: 

(1.3)  Mean  square  error  (MSE)  =  <Jy2  +  e2, 

where  (Ty2  is  the  variance  of  the  mean  of  the  observed  values,  yi,  and  e  is 
the  bias  in  the  estimate  of y. 

Clearly,  whenever  bias  is  present  the  variance  of  the  sample  estimate 
is  increased,  and  the  more  so  as  the  square  of  the  bias  rises  relative  to  the 

14  In  many  surveys,  particularly  those  dealing  with  opinions  or  attitudes,  "true 
value"  may  be  a  nebulous  concept.  In  the  present  case,  we  may  assume  that  we 
are  dealing  with  factual  information  (such  as  face  value  of  life  insurance)  and  that 
the  "true  value"  refers  to  the  actual  amount  held  by  that  sample  member.  Ad- 
mittedly, even  in  such  a  situation  questions  may  arise  regarding  "true  value,"  as  in 
the  case  of  equity  in  a  pension  fund,  but  this  is  beyond  the  scope  of  the  present 
discussion. 

"  For  example,  W.  G.  Cochran,  Sampling  Techniques  (New  York:  Wiley, 
1963),  pp.  12-16. 
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variance  of  the  observations.   Thus,  if  e  is  a  certain  multiple  of  (Ty,  say  k, 
the  above  equation  reduces  to: 

(1.4)  MSE  =  (1  +  *2)  <rv2. 

Some  idea  of  the  extent  of  this  bias  can  be  obtained  by  considering 
the  manner  in  which  different  values  of  k  influence  the  probability  that 
the  usual  symmetrical  95  percent  confidence  interval  will  include  the 
true  mean.  The  exact  probability  in  the  absence  of  bias,  using  both  tails 
of  the  normal  distribution,  is  .95.  The  following  tabulation  gives  values 
of  this  probability  for  different  values  of  k.16 

Value  of  k  Probability 

0 950 

.25 941 

.5 921 

.75 883 

1.0 830 

1.5 677 

2.0 484 

3.0 149 

4.0 021 

5.0 001 

For  relatively  small  biases  —  even  those  in  which  the  bias  is  50  per- 
cent of  the  variance  of  the  true  value  —  little  distortion  takes  place  in  the 
standard  error  estimate  and  in  the  confidence  interval.  However,  as 
the  bias  approaches,  and  then  surpasses,  the  standard  error  of y,  the  bias 
increases  at  an  increasing  rate.  When  the  bias  term  is  more  than  twice 
the  size  of  the  standard  error  of  y,  the  probability  that  the  confidence 
interval  contains  the  true  mean  is  just  about  half  the  presumed  value; 
and  if  the  bias  exceeds  the  standard  error  of  y  by  more  than  4.0,  the 
probability  is  virtually  zero. 

Equation  (1 .3)  also  may  be  used  to  demonstrate  that  even  in  the  absence 
of  bias  the  estimated  variance  of  the  mean  may  differ  from  the  true  vari- 
ance as  long  as  errors  are  present  in  the  individual  observations.  The 
reason  for  this  is  that  the  variance  term  on  the  right-hand  side  of  Equation 
(1.3)  is  based  on  the  observed  values,  yi,  not  on  the  true  values,  #».  As  a 
result,  using  the  definition  in  Equation  (1.1),  it  is  a  simple  matter  to  show 
that  (Jy2  represents  a  combination  of  sampling  variances  of  *;  and  of  d, 
as  follows: 

(1 .5)  (Ty2  =  (Tr  4-  OV2  -f  2r(Tx<T7, 
where  r  is  the  coefficient  of  correlation  between  xt  and  a. 

16  Adapted  from  Cochran,  ibid.   The  probability  is: 

.  rlM+ela- 

L=-/  -£. 

v  J-1.96-e/<T 
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Equation  (1 .5)  reflects  the  errors  in  the  individual  observations.  Hence, 
to  arrive  at  the  total  effect  of  nonsampling  errors  on  the  reliability  of 
survey  results,  we  may  substitute  Equation  (1.5)  into  Equation  (1.3)  to  ob- 
tain: 

(1 .6)  MSE  =  a-2  +  (Te2  +  2r<Tx(Te  +  e\ 

The  effect  on  the  sampling  variance  of  nonsampling  errors  is  measured 
by  the  last  three  terms  on  the  right-hand  side.  The  last  term  measures 
the  increase  in  the  variance  due  to  bias,  whereas  the  preceding  two  terms 
measure  the  increase  in  the  variance  due  to  errors  in  the  individual 
observations,  including  intercorrelation  between  these  errors  and  the 
true  values. 

Equation  (1 .6)  serves  as  a  framework  for  the  analysis  of  the  effect  of 
nonsampling  errors  on  the  reliability  of  survey  results.  As  has  been 
demonstrated,  this  effect  is  manifested  in  two  ways:  through  increase 
in  the  sampling  variance  of  the  observations,  because  this  variance  is 
based  on  yi  rather  than  x%,  and  through  distortion  of  the  variance  if  bias 
is  present,  because  the  sampling  variance  is  estimated  around  y,  not  the 
unbiased  value  x.  To  be  sure,  the  former  effect  is  incorporated  in  the 
estimate  of  the  sampling  variance  and  therefore  does  not  bias  this  estimate. 
However,  to  the  extent  that  this  component  can  be  eliminated,  the  re- 
liability of  the  estimate  of  the  sampling  variance  is  increased. 

The  effect  of  bias  is  measured  by  the  ratio,  e/av,  or  k  (Equation  1.4). 
As  k  increases,  the  apparent  sampling  variance,  <Ty2,  differs  increasingly 
from  the  true  sampling  variance,  or  mean  square  error.  With  validation 
data,  these  quantities  can  be  computed  and  compared  with  each  other. 

In  a  similar  fashion,  the  effect  of  errors  in  individual  observations  can 
be  computed  by  comparing  dy2  with  cr^2,  or: 

(1.7)  Relative  increase  in  sampling  variance  due  _    (Te2  +  2rO"50'? 

to  errors  in  individual  observations  oy* 

The  validation  data  enable  us  not  only  to  compute  this  ratio  but  also 
to  allocate  the  increase  among  the  two  principal  causes:  errors  and  inter- 
correlation between  errors  and  true  values. 

Nonsampling  Errors  in  Savings  Surveys 

Different  types  of  surveys  are  subject  to  nonsampling  errors  in  varying 
degrees,  depending  on  certain  salient  characteristics  of  the  surveys  and  of 
the  populations  sampled.  Consumer  financial  surveys  possess  distinctive 
characteristics  which  contribute  to  the  likelihood  of  more  frequent  occur- 
rences of  certain  types  of  nonsampling  errors  than  are  found  on  other 
surveys.  A  review  of  the  main  such  characteristics  and  their  implications 
for  the  nonsampling-error  problem  would  seem  to  be  in  order.  For  this 
purpose,    we    distinguish    between    "random"    and    "nonrandom"    non- 
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sampling  errors.  Random  nonsampling  errors  are  defined  as  those  which 
are  scattered  symmetrically  about  the  true  values,  so  that  even  though 
single  deviations  may  be  large,  on  an  overall  basis  the  deviations  tend  to 
nullify  each  other  as  the  sample  size  increases.  In  other  words,  the  ex- 
pected value  of  the  observations  converges  stochastically  to  the  true  value. 
On  the  other  hand,  nonrandom  sampling  errors  tend  to  bias  the  observed 
value  away  from  the  true  value. 

For  analytical  work  — ■  establishing  relationships  between  variables  — 
both  types  of  errors  may  be  of  importance,  the  primary  determinant  being 
their  size  and  dispersion  rather  than  their  randomness  or  lack  of  random- 
ness. However,  for  enumerative  purposes  —  estimating  averages  or  totals 
—  it  is  essentially  the  nonrandom  errors  that  matter,  for  with  increasing 
sample  size  the  effect  of  the  random  nonsampling  errors  will  tend  to 
cancel  out. 

The  distinction  between  random  and  nonrandom  nonsampling  errors 
is  not  always  clear-cut  and  may  vary  with  what  is  being  measured.  Thus, 
response  errors  may  be  random  for  estimation  of  debt  but  nonrandom  for 
estimation  of  stock  holdings.  In  a  particular  case,  however,  the  distinction 
generally  can  be  made,  sometimes  on  an  a  priori  basis  and  at  other  times 
on  the  basis  of  past  experience. 

Personal  Nature  of  Interview 

In  a  savings  interview  of  the  type  considered  here,  the  respondent 
is  questioned  among  other  things  about  either  his  saving  during  a  specific 
period  (the  last  month,  the  last  quarter,  or  the  last  year  being  the  most 
common  periods)  or  about  his  savings  at  a  particular  moment  of  time, 
often  as  of  the  date  of  the  interview.  In  either  case,  the  questions  are  of 
highly  personal  nature,  delving  into  matters  which  the  respondent  may 
have  told  nobody  (not  even  his  psychiatrist  .  .  .)  and,  accordingly, 
raising  stresses  and  strains  not  ordinarily  present  in  survey  interviews. 

In  addition,  the  presence  of  an  interviewer,  although  not  usually  from 
the  same  neighborhood  but  invariably  from  the  same  city  or  county,  is 
likely  to  place  the  respondent  under  various  social  pressures,  the  nature 
of  these  pressures  depending  on  his  personality.  Thus,  if  the  respondent 
is  a  conformist,  he  will  hesitate  to  give  information  showing  lack  of  con- 
formity with  the  group  to  which  he  feels  a  kinship,  such  as  admitting  hav- 
ing more  liabilities  than  assets  while  living  in  a  very  prosperous  neighbor- 
hood. If  he  is  on  relief,  he  may  not  care  to  divulge  ownership  of  a  sizeable 
savings  account  for  fear  of  losing  his  relief  status  if  this  fact  should  become 
known  to  the  proper  authorities.  Alternatively,  he  may  not  care  to  reveal 
sizeable  holdings  of  stocks  and  bonds  while  living  in  a  poor  neighborhood, 
because  of  not  having  reported  the  income  from  these  holdings  on  his  tax 
returns,  or  for  other  reasons. 
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Whatever  the  nature  of  these  pressures,  all  of  them  contribute  to 
increased  incidence  of  response  errors.  This  is  particularly  true  in  view 
of  the  ease  of  making  such  errors.  The  answers  can  hardly  be  checked, 
and  if  a  respondent  is  not  sure  of  an  answer,  or  does  not  want  to  give  the 
correct  answer,  it  is  a  simple  matter  for  him  to  give  some  other  answer. 
Alternatively,  he  may  refuse  altogether,  which  is  another  form  of  response 
error  (unless  the  entire  interview  is  refused,  in  which  case  it  becomes  a 
form  of  noncontact) . 

All  of  these  errors  are  likely  to  be  serious,  since  not  only  can  they  be 
substantial  but  they  are  without  any  doubt  nonrandom.  The  combination 
of  social  pressures  and  personal  stress  brought  about  by  the  questions  on 
savings  or  saving  can  lead  to  substantial  overestimates  and  to  substantial 
underestimates,  with  the  latter  tending  to  be  considerably  larger.  As  we 
shall  see,  the  most  important  response  error,  or  nonsampling  error  of  any 
type  on  a  consumer  savings  survey,  is  a  report  of  no  holdings  when  an 
asset  is  owned.  The  personal  nature  of  the  interview  also  increases  the 
frequency  of  noncontact  errors,  and  it  also  contributes  to  their  being 
primarily  nonrandom.  Evidence  will  be  presented  later  to  show  that 
people  who  refuse  to  be  interviewed  or  are  not  contacted  tend  to  differ 
from  respondents  in  certain  significant  respects. 

Request  for  Figures 

The  request  for  exact  figures,  generally  unavoidable,  is  another  cause 
of  nonsampling  error,  again  a  contribution  to  response  error.  Whether 
the  request  is  for  a  balance  or  for  a  change  during  a  given  period,  the 
exact  figure  is  hardly  at  the  respondent's  fingertips,  and  there  is  a  strong 
inclination  to  cite  a  figure  from  memory  rather  than  to  start  searching 
for  records.  Clearly,  the  inclination  is  even  stronger  if  the  respondent  is 
not  in  a  fully  cooperative  mood. 

Quoting  from  memory  figures  that  in  many  cases  relate  to  matters 
with  which  the  respondent  may  not  have  had  recent  contact  no  doubt 
serves  to  contribute  to  the  incidence  of  response  errors,  both  directly  and 
through  possible  interaction  with  the  preceding  factor.  Thus,  if  a  respon- 
dent is  somewhat  dubious  about  mentioning  the  complete  extent  of  his 
life  insurance  holdings,  this  tendency  may  manifest  itself  through  a 
memory  quirk  leading  him  to  report,  in  all  sincerity,  a  lower  amount  of 
holdings  than  is  actually  the  case. 

Complicated  Subject  Matter 

Still  another  factor  contributing  to  response  error  in  savings  surveys 
is  the  complexity  of  the  subject  matter.  Even  when  people  have  savings 
in  a  particular  form,  they  may  not  know  too  well  what  it  is.  Savings  and 
loan  associations  tend  to  be  confused  with  mutual  savings  banks,  and  both 
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with  commercial  banks;  investment  accounts  with  savings  share  accounts; 
common  stock  with  preferred  stock;  term  insurance  with  ordinary  life 
insurance;  and  so  on.  At  times  neither  the  respondent  nor  the  interviewer 
may  be  too  familiar  with  a  particular  type  of  asset,  a  situation  which  un- 
doubtedly increases  the  incidence  of  errors  brought  about  by  faulty  re- 
cording or  misinterpretation. 

Insecurity  resulting  from  respondent  awareness  of  his  own  limitations 
in  dealing  with  questions  on  this  subject  may  also  contribute  to  refusals 
and  noncontacts.  This  is  particularly  likely  if  the  respondent  knows  in 
advance  that  fairly  detailed  questions  will  be  asked  about  his  financial 
position  or  about  his  knowledge  of  different  assets  and  debts.  (This  is 
one  reason  why  introductory  letters  are  best  made  brief  and  not  too 
explicit.)  Furthermore,  because  of  the  complex,  and  confidential,  nature 
of  the  subject,  the  interview  must  be  conducted  with  a  family  member  who 
is  acquainted  with  its  finances,  not  just  with  any  member  that  happens  to 
be  at  home.  This,  of  itself,  makes  a  refusal  or  noncontact  more  likely,  for 
if  that  particular  family  member  is  unavailable  for  an  interview,  there  is 
no  alternative. 

To  a  lesser  extent,  complexity  of  subject  matter  contributes  to  the 
incidence  of  data-processing  errors,  particularly  in  transcribing  hurriedly- 
written  dollar  amounts.  As  a  rule,  however,  such  errors  are  likely  to  be 
random  and  can  be  reduced  to  negligible  proportions  with  provisions  for 
check-coding  and  data  verification. 

Length  of  Interview 

Few  interviews  on  one's  savings  habits  and  practices  take  less  than 
half  an  hour;  some  take  almost  two  hours.  Although  these  are  by  no 
means  the  longest  duration  of  personal  interviews,17  toward  the  end  of 
such  interviews  — ■  particularly  after  the  first  hour  —  some  respondents 
tend  to  become  restless  and  try  to  finish  the  interview  as  quickly  as  pos- 
sible. Under  such  circumstances,  response  errors  can  be  more  frequent, 
especially  errors  on  dollar  amounts  and  other  factual  data  about  which 
the  respondent  may  not  then  want  to  devote  much  thought. 

On  lengthy  interviews,  errors  also  become  more  likely  from  the  inter- 
viewer's side,  for  he  too  may  be  fatigued,  less  alert,  and  more  prone  to 
misinterpret,  to  record  answers  incorrectly,  and  even  to  abbreviate  ques- 
tions to  the  point  of  confusing  the  respondent. 

Highly  Skewed  Population 

Economic  and  financial  data  distributions  are  generally  highly  skewed 

17  The  consumer  expenditure  interviews  of  the  United  States  Bureau  of  Labor 
Statistics  may  extend  over  eight  hours,  although  not  usually  continuously;  and 
other  interviews  on  consumer  expenditures  requiring  anywhere  between  two  and 
four  hours  are  not  uncommon. 
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to  the  right,  and  this  is  especially  true  of  savings  and  saving.  As  shown  by 
Lampman,18  in  1953  the  great  majority  of  American  consumer  units  held 
relatively  little  wealth,  and  the  situation  has  changed  little  since  that  time. 
At  one  end  of  the  scale,  31  percent,  holding  less  than  $1,000  of  assets  per 
consumer  unit,  accounted  for  1  percent  of  the  nation's  total  financial 
wealth.  This  is  no  doubt  at  least  as  true  of  saving  during  a  given  period. 
The  proportion  of  consumer  units  adding,  say,  more  than  $100,000  to 
their  savings  accounts  or  to  their  stocks  during  a  given  period  is  un- 
doubtedly extremely  low,  yet  of  substantial  importance  to  the  estimation 
of  aggragate  saving.  As  a  result,  unless  the  sample  contains  many  thou- 
sands of  consumer  units  —  and  most  surveys  of  consumer  savings  rarely 
cover  more  than  3,000  consumer  units  —  errors  relating  to  the  sampling 
distribution  are  not  unlikely,  and  they  are  clearly  nonrandom. 

Such  errors  are  much  less  likely  for  estimation  of  attributes,  e.g.,  the 
proportion  of  consumer  units  increasing  their  assets  by  more  than  a  cer- 
tain amount,  rather  than  for  dollar  amounts.  It  is  for  this  reason  that  the 
former  type  of  estimation  is  generally  preferred  when  a  choice  is  possible. 
Even  in  such  cases,  however,  errors  relating  to  the  sampling  distribution 
can  be  expected,  since  a  difference  of  one  or  two  sample  members  in  an 
extreme  class  can  amount  to  many  thousands  of  consumer  units  when 
aggregated. 

Estimation  of  Unknown  Information 

Complete  coverage  of  saving,  even  if  only  financial  saving,  leads  to 
requests  for  data  about  which  a  respondent  can  hardly  be  sure  of  being 
correct.  Saving  in  the  form  of  pensions  is  a  case  in  point.  Theoretically, 
once  the  concept  of  saving  in  this  form  is  defined,  the  necessary  informa- 
tion could  be  obtained  either  from  the  respondent's  records  or  by  going 
to  the  institutional  source.  In  practice,  however,  such  records  are  rarely 
available  and  it  is  not  usually  feasible  to  obtain  the  data  from  the  institu- 
tion. The  result  is  that  peripheral  data  are  generally  sought  from  the 
respondent,  and  the  data  then  serve  as  a  basis  for  estimating  saving  in 
the  form  of  pensions. 

These  estimates  are  subject  to  errors  of  valuation,  stemming  from  two 
sources.  First,  the  peripheral  information  supplied  by  the  respondent  may 
not  be  accurate,  in  which  case  even  a  fully  accurate  estimation  procedure 
will  lead  to  some  error.  Second,  the  estimation  procedure  will  very  likely 
not  be  perfect.  Although  it  may  be  stochastic,  so  that  the  average  of 
many  individual  estimates  will  tend  to  coincide  with  the  true  average, 
errors  in  individual  cases  may  well  be  substantial.    If  the  estimation  pro- 

18  Robert  Lampman,  "Changes  in  the  Share  of  Wealth  Held  by  Top  Wealth 
Holders,"  Review  of  Economics  and  Statistics,  Vol.  41,  No.  4  (November,  1959), 
pp.  379-92. 
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cedure  has  been  well  designed,  these  errors  will  tend  to  be  random.  The 
first  type  of  error  is  less  likely  to  be  random,  however,  to  the  extent  to 
which  personal  data  are  requested,  such  as  contributions  to  a  pension 
plan.   The  exact  nature  of  these  errors  is  as  yet  unknown. 

If  saving  in  the  form  of  durable  or  other  tangible  goods  is  included, 
the  scope  for  errors  of  valuation  increases  enormously,  for  reasons  already 
discussed.  Hopefully,  such  errors  are  random,  although  their  magnitude 
is  likely  to  be  substantially  greater  than  errors  of  valuation  of  financial 
assets. 

Overall  View 

This  brief  review  of  distinctive  characteristics  of  consumer  saving 
surveys  is  sufficient  to  bring  out  clearly  the  fact  that  the  danger  of  re- 
sponse and  noncontact  errors  is  greatly  increased  in  such  surveys.  The 
increased  incidence  and  seriousness  of  these  nonsampling  field  errors  is 
essentially  the  direct  effect  of  the  complexity  and  distinctive  nature  of 
these  interviews.  Partly  as  a  result,  other  types  of  nonsampling  errors, 
such  as  data-processing  errors,  tend  to  increase  also,  although  undoubtedly 
nowhere  near  to  the  extent  that  errors  in  the  field  increase.  Although 
exact  estimates  of  the  relative  magnitudes  of  these  different  types  of  non- 
sampling  errors  have  never  been  made,  there  is  little  doubt  that  at  least 
in  consumer  financial  surveys  these  field  errors  are  by  far  the  more  im- 
portant, and  that  adequate  means  of  coping  with  them  would  contribute 
more  than  any  other  single  factor  toward  raising  the  reliability  of  such 
surveys. 

Panel  Effects 

Although  panels  offer  unique  advantages  for  the  study  of  consumer 
saving  behavior,  they  also  serve  to  introduce  additional  ramifications  to 
the  nonsampling  error  problem.  We  shall  discuss  these  sources  with 
specific  reference  to  the  type  of  panel  used  in  the  course  of  the  Consumer 
Savings  Project.  In  these  panels  the  same  family  is  interviewed  approxi- 
mately every  three  months  for  at  least  one  year.  In  each  interview  data 
are  sought  regarding  saving  in  the  intervening  period  or  the  stock  of 
savings  at  the  time  of  the  interview.  Additional  questions  are  also  asked, 
differing  from  one  interview  to  another.  These  questions  may  be  designed 
partly  to  acquire  supplementary  information  —  income,  saving  attitudes, 
savings  practices,  and  so  forth  —  and  partly  to  improve  rapport  by  reduc- 
ing the  apparent  emphasis  on  securing  dollar  figures. 

The  additional  sources  of  nonsampling  errors  stem  primarily  from  the 
repetitive  interviewing  of  the  same  people,  intensified  in  a  consumer 
savings  panel  by  the  fact  that  the  same  basic  questions  are  repeated  from 
one  interview  to  another.   On  the  other  hand,  this  repetitive  interviewing 
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procedure  may  also  contribute  to  reducing  certain  nonsampling  errors, 
as  will  be  noted  shortly.  First,  however,  let  us  consider  the  possible 
deleterious  effects  of  a  panel  operation.  These  effects  relate  either  to 
possible  loss  of  representativeness  of  the  panel  over  time  or  to  conditioning 
effects  due  to  respondent  participation. 

Representativeness 

Whether  or  not  a  panel  provides  a  perfect  miniature  of  the  population 
at  the  start  of  the  operation,  chances  are  that  it  will  become  less  repre- 
sentative as  time  goes  on,  for  two  reasons.  First,  a  panel  is  a  static  minia- 
ture: only  to  a  limited  extent,  if  at  all,  will  it  reflect  changes  in  popula- 
tion characteristics  over  time.  Migration,  new  family  formation,  and 
family  dissolution  are  among  the  demographic  factors  that  cannot  be 
replicated  by  a  static  panel. 

To  be  sure,  attempts  can  be  made  to  introduce  such  autonomous 
factors  through  a  policy  of  panel  rotation  —  periodic  replacement  of  old 
panel  members  with  new  ones,  with  appropriate  stratification  to  allow 
for  changing  population  composition.  Such  a  policy,  however,  reduces 
the  period  over  which  data  are  available  for  individual  families.  This 
policy  is  followed  by  various  commercial  consumer  panels,  in  which  data 
are  sought  for  individual  families  over  long  periods  of  time.  It  has  not 
been  used,  as  far  as  is  known,  in  consumer  saving  panels,  which  are 
generally  of  brief  enough  duration  that  changes  in  population  composi- 
tion assume  negligible  importance. 

The  second  reason  for  possible  lack  of  representativeness  is  far  more 
serious  for  a  savings  panel,  and  probably  for  other  continuous  panels  as 
well,  namely,  panel  mortality  and  nonresponse.  Since  a  panel  is  essen- 
tially an  extension  of  a  one-time  survey,  the  refusal  and  noncontact  rates 
on  the  first  interview,  or  wave,  of  a  panel  operation  are  certainly  no  less 
than  those  encountered  in  a  corresponding  one-time  survey  interview  — 
and  may  be  considerably  larger  if  sample  members  are  informed  of  the 
panel  nature  of  the  operation  before  the  interview  is  carried  out.  On  the 
second  wave,  nonresponse  may  be  less  than  on  the  first  wave,  but  it  still 
tends  to  be  sizeable,  perhaps  because  only  then  do  some  respondents  be- 
come fully  cognizant  of  the  meaning  of  the  panel  operation.  Nonresponse 
on  later  waves  usually  declines  sharply,  although  rarely  to  zero,  because 
by  that  time  the  panel  consists  essentially  of  a  "hard  core"  of  cooperative 
respondents,  at  least  so  far  as  willingness  to  grant  an  interview  is  con- 
cerned.19 


19  Nonresponse  rates  in  panel  operations  of  the  Consumer  Savings  Project  will 
be  presented  in  Chapter  VI.  For  a  discussion  of  nonresponse  rates  in  a  different 
type  of  panel,  see  Robert  Ferber,  "Observations  on  a  Consumer  Panel  Operation," 
Journal  of  Marketing,  Vol.  17,  No.  3  (January,  1953),  pp.  246-59. 
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Those  dropping  out  of  the  panel  do  so  for  a  number  of  reasons  —  sick- 
ness or  death,  moving  away,  dislike  of  the  interviewer,  boredom  with  the 
study,  resentment  of  questions,  and  so  on.  Taken  as  a  group,  however, 
there  is  always  doubt  as  to  whether  these  people  are  similar  in  the  charac- 
teristics under  study  to  those  who  remain  in  the  panel,  particularly  in  the 
case  of  a  savings  panel.  From  a  survey  viewpoint  these  dropouts  consist 
of  two  groups:  the  noncontacts  and  the  refusals.  Noncontacts  are 
generally  people  who  are  not  at  home.  In  addition,  in  a  savings  survey 
they  are  likely  to  include  very  high-income,  high-asset  sample  members  — 
people  who  exert  tremendous  influence  on  the  dollar  volume  of  savings 
transactions  and  who,  partly  for  this  reason,  are  difficult  to  contact.  In  a 
savings,  or  any  economic,  panel,  dropouts  are  also  likely  to  include  rela- 
tively high  proportions  of  independent  businessmen.  Members  of  both  of 
the  latter  groups  may  be  willing  to  give  one  or  two  interviews  at  the 
start.  However,  partly  because  of  pressures  of  time  and  partly  perhaps 
because  they  may  be  more  reluctant  to  discuss  their  financial  position,  they 
would  seem  more  likely  than  other  people  to  drop  out  of  a  panel  opera- 
tion (as  will  be  corroborated  in  Chapter  VI) . 

Similar  inferences  can  be  drawn  about  refusals  in  a  savings  panel. 
Refusals,  more  likely  than  noncontacts,  will  include  people  who  do  not 
care  to  discuss  their  finances,  having  perhaps  taken  some  action  in  the 
recent  past,  or  possibly  planning  to  take  one,  that  influences  them  to  dis- 
continue further  participation.  They  are  also  very  likely  to  be  people  for 
whom  time  is  at  a  premium,  and  hence  they  are  probably  in  high-income 
brackets  or  have  their  own  business. 

Progressive  dropping  out  of  people  of  this  type  is  undoubtedly  the 
principal  cause  of  lack  of  representativeness  in  a  savings  panel,  a  type  of 
nonsampling  error  which  is  clearly  nonrandom  and  can  increase  with 
time.  Item  nonresponse  may  be  a  contributory  factor:  there  is  little 
doubt  that  respondents  refusing  to  answer  individual  questions  probably 
differ  from  other  respondents,  but  it  is  not  clear  if  this  atypicalness  in- 
creases with  time. 

Conditioning  Effects 

Does  participation  in  a  panel  operation  cause  respondents  to  act  or 
respond  differently  from  what  they  normally  would?  If  so,  another  form 
of  nonsampling  error  has  to  be  dealt  with. 

The  basis  for  expecting  an  error  of  this  form  is  that  presumably  panel 
members  tend  to  become  more  conscious  of  their  participation  in  the 
panel  as  time  goes  on,  which  manifests  itself  in  altered  attitudes  or  be- 
havior. The  resulting  information  supplied  by  these  people  is  then 
atypical  of  the  population  group  which  they  originally  represented,  and 
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thus  the  nonsampling  error  is  more  likely  than  not  to  be  nonrandom  as 
well. 

Conditioning  effects  of  this  type  are  considered  most  likely  in  attitu- 
dinal  studies,  particularly  studies  relating  to  personal  habits  or  to  questions 
of  social  import.  Thus,  a  panel  operation  on  budgetary  practices  and 
durable  goods  purchases  may  induce  participants  to  become  more  con- 
scious of  the  desirability  of  budgeting,  with  the  result  that  some  of  them 
begin  to  budget  purchases  largely  as  a  consequence  of  their  membership 
in  the  panel. 

The  extent  to  which  conditioning  effects  may  exist  in  a  particular  case 
depends  on  an  evaluation  of  many  facets  of  the  operation  and  will  require 
much  experimentation  before  a  conclusive  judgment  can  be  made.  Any- 
thing likely  to  affect  the  respondent's  replies  has  to  be  considered  —  the 
content  of  the  questionnaire,  the  frequency  of  interviews,  the  approach 
taken  by  the  interviewer  (if  one  is  used),  the  contents  of  any  advance  or 
introductory  letter,  information  supplied  to  the  respondent  relating  to  the 
survey,  and  so  forth.  If  the  presence  of  conditioning  effects  is  suspected, 
about  the  only  way  of  proving  it  is  to  introduce  new  members  into  the 
panel  periodically,  ask  the  new  and  the  old  members  the  same  questions, 
and  then  compare  the  results.  In  a  large  continuing  panel  operation  this 
is  not  difficult  to  do,  particularly  since  it  is  also  a  means  of  coping  with 
the  problem  of  representativeness,  as  was  noted  earlier.20 

Whether  conditioning  effects  are  likely  to  be  present  in  a  consumer 
savings  panel  of  the  type  under  discussion  here  is  not  clear.  Since  the 
focus  of  the  operation  is  on  saving  and  on  saving  habits,  the  attention  of 
panel  members  is  centered  on  this  subject  much  more  than  would  other- 
wise be  the  case.  Certainly  they  are  made  more  aware  of  saving  alterna- 
tives and  reasons  for  saving.  But  does  this  influence  them  to  save  more 
(or  less)  or  to  invest  their  savings  otherwise  than  they  ordinarily  would 
have?  Saving  habits  would  seem  too  deeply  ingrained  to  be  affected 
appreciably  by  one  interview  on  the  subject  every  three  or  four  months 
for  a  year.  To  be  sure,  there  is  always  the  possibility  that  the  interview 
will  remind  a  respondent  of  an  action  he  had  been  planning  to  take  and 
hence  lead  him  to  it,  e.g.,  to  convert  an  ordinary  life  policy  to  an  endow- 
ment policy.  Data  presented  in  Chapter  VI  suggest  that  to  the  extent 
that  such  actions  did  take  place  they  were  not  of  major  importance,  con- 
sidering the  panel  as  a  whole.  The  fact  remains,  however,  that  condition- 
ing effects  are  an  ever-present  danger  and  that  rotating  panel  member- 
ship is  to  be  highly  recommended  in  any  continuing  panel  operation. 

20  For  an  example  of  this  procedure,  see  Joseph  Waksberg  and  R.  P.  Pearl, 
"The  Current  Population  Survey:  A  Case  History  in  Panel  Operations,"  Proceed- 
ings of  the  Social  Statistics  Section  of  the  American  Statistical  Association  (Wash- 
ington:   American  Statistical  Association,  1964),  pp.  217-29. 
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Restriction  on  the  length  of  panel  membership  can  be  a  positive  force  in 
maintaining  panel  member  rapport  and,  as  was  noted  earlier,  serves  to 
mitigate  both  principal  sources  of  nonsampling  error. 

Beneficial  Effects 

Although  not  undertaken  primarily  for  this  purpose,  a  panel  operation 
does  offer  interesting  possibilities  for  the  reduction  of  nonsampling  errors. 
The  continued  contact  with  the  panel  members  offers  excellent  opportu- 
nities for  reconciling  answers  that  are  inconsistent,  unclear,  or  possibly 
based  on  misinterpretation  of  the  questions.  To  be  sure,  something  along 
the  same  line  can  be  done  in  a  one-time  survey  by  means  of  callbacks  and 
supplementary  letters,  but  it  is  hardly  likely  to  be  done  to  anywhere  nearly 
so  great  an  extent  or  with  the  degree  of  economy  with  which  it  can  be 
done  in  a  panel  study  —  and  generally  it  is  not  done  at  all.  For  studies 
in  which  extensive  data  collection  is  involved  and  errors  of  interpretation 
or  transcription  are  not  unlikely,  this  checking  process  can  be  very  useful. 

Improved  rapport  leading  to  the  supplying  of  more  complete  and 
more  accurate  data  is  another  major  advantage  of  a  panel  operation  with 
regard  to  reducing  nonsampling  errors.  This  is  a  particular  advantage 
in  a  savings  panel,  in  which  the  personal  nature  of  the  questions  leads 
some  panel  members  to  fear  ulterior  motives,  such  as  secret  participation 
by  the  Internal  Revenue  Service.  In  such  instances,  only  after  two  or 
three  interviews  is  full  cooperation  likely  to  be  obtained  and  complete 
data  made  available.  At  the  same  time,  there  are  instances  in  which  the 
reverse  occurs,  when  rapport  deteriorates  possibly  to  a  point  at  which 
the  panel  member  drops  out. 

Apart  from  rapport,  a  major  reason  for  improvement  lies  in  the 
increased  accuracy  with  which  data  are  recalled  on  later  interviews.  This 
was  particularly  noticeable  in  a  durable  goods  consumer  purchase  panel 
conducted  some  time  ago,  in  which  the  average  number  of  purchases 
reported  increased  consistently  with  length  of  panel  membership.21 
Membership  on  the  panel,  combined  with  advance  notice  of  an  inter- 
viewer visit,  apparently  motivated  the  panel  members  to  recall  more  com- 
pletely durable  goods  purchases  made  since  the  preceding  interview. 

This  factor  is  undoubtedly  operative  with  a  consumer  savings  panel, 
even  though  panel  members  are  continually  asked  to  consult  records 
rather  than  to  cite  figures  from  memory.  Knowing  this  fact  in  advance 
tends  to  make  records  either  more  readily  available  or  to  give  the  respond- 
ent a  chance  to  look  up  the  figures  in  his  records.  Here  again,  the  effect 
could  be  in  the  reverse  direction,  but  this  appears  to  be  more  the  excep- 
tion than  the  rule. 

21  Ferber,  op.  cit. 
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From  an  overall  point  of  view,  perhaps  the  primary  beneficial  effect 
of  a  panel  operation  in  dealing  with  nonsampling  errors  —  in  this  case, 
response  and  noncontact  errors  —  is  to  bring  the  problem  more  sharply 
into  focus.  With  refusals  and  noncontacts,  information  is  invariably  avail- 
able from  prior  waves  to  serve  as  a  basis  for  a  sound  judgment  of  their 
effects.  Similarly,  the  wealth  of  previously  obtained  information  on  panel 
members  makes  it  much  easier  to  detect  response  errors  and  even  to  pin- 
point their  causes.  Despite  deterioration  of  rapport  with  some  panel  mem- 
bers, the  overall  result  turns  out  to  be  more  accurate  data  both  on  a 
current  and  on  an  ex  post  basis. 

General  Comments 

The  foregoing  review  suggest  that  the  data  reliability  problem  is  far 
more  serious  in  a  consumer  savings  survey  than  it  is  in  most  other  types 
of  surveys  and  even  in  most  other  economic  surveys.  The  problem  is  one 
essentially  of  coping  with  nonsampling  errors,  particularly  with  response 
and  noncontact  errors.  The  latter  types  of  error  can  be  tremendously 
more  frequent  in  savings  surveys,  producing  serious  biases  and  distortions 
in  the  survey  data.  For  documentation  of  these  tendencies,  we  turn  to  a 
consideration  of  the  available  studies. 


II.  AGGREGATIVE  COMPARISONS 


This  chapter  presents  the  few  comparisons  made  of  consumer  survey 
derived  aggregates  for  components  of  savings  or  saving  with  correspond- 
ing aggregates  obtained  from  other  sources.  Since  the  accuracy  of  the 
latter  aggregates  is  generally  not  known,  the  results  of  such  comparisons 
have  to  be  interpreted  with  considerable  caution.  In  particular,  they  can- 
not indicate  the  precise  degree  of  error  in  the  survey  data,  and  only  in  the 
most  indirect  fashion  can  they  point  to  possible  causes  or  reasons  for  error. 
They  only  serve  to  indicate  the  presence  of  error  to  the  extent  that  the 
aggregates  disagree  and  it  is  not  likely  to  be  clear  which  aggregate  is  in 
error  by  how  much.  Nevertheless,  as  we  shall  see,  useful  inferences  are 
possible. 

Detailed  attempts  to  aggregate  survey  data  on  savings  or  on  saving 
have  as  yet  been  made  only  in  the  United  Kingdom  and  in  the  United 
States.  Accordingly,  this  chapter  focuses  on  such  studies  conducted  thus 
far  in  these  two  countries. 

United  Kingdom  Savings  Surveys 

Savings  surveys  were  carried  out  in  the  United  Kingdom  during 
1952-55.  A  primary  objective  of  these  surveys  was  to  obtain  data  on  asset 
holdings  and  on  saving  in  various  forms  between  the  time  of  one  survey 
and  that  of  the  next.  It  is  therefore  of  particular  interest  to  examine  how 
well  the  survey  estimates  of  these  aggregates  compare  with  those  from 
other  sources.  Such  a  comparison  is  shown  for  total  liquid  assets  and  for 
the  two  primary  categories  of  English  liquid  assets  in  Table  1.  The  defi- 
nition of  liquid  assets  in  the  United  Kingdom  is  essentially  the  same  as  the 
definition  used  in  this  volume,  including  individuals'  holdings  of  checking 
accounts,  savings  accounts  in  the  various  types  of  savings  institutions,  and 
British  government  bonds;  holdings  of  currency,  however,  are  excluded. 

Three  sets  of  survey  estimates  are  shown  in  Table  1  —  estimates  based 
on  the  survey  data  alone,  estimates  based  on  weighting  the  survey  re- 
sponses by  income  level  with  independent  estimates  of  the  British  income 
distribution  computed  from  tax  data,  and  estimates  obtained  by  weighting 

25 
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Table  1.  Comparison  of  Survey  Estimates  of  Aggregate 

Liquid  Asset  Holdings  in  United  Kingdom  with 

Independent  Estimates.,  1952-55 


Date 

Independent 

estimate 

(millions 

of  pounds) 

Survey  estimate  as  a  percentage 
of  independent  estimate 

Category 

Un- 
adjusted 

Weighted 
by  income 

Weighted 
by  occu- 
pation 

"National 

Savings"" 

Other  liquid 

assetsb 

April,  1952 
April,  1953 
April,  1954 
June,  1955 

April,  1952 
April,  1953 
April,  1954 
June,  1955 

April,  1952 
April,  1953 
April,  1954 
June,  1955 

5,413 
5,316 
5,292 
5,408 

3,368 
3,577 
3,810 
4,037 

8,781 
8,893 
9,102 
9,445 

62 
56 
59 
64 

61 

47 
77 
67 

61 
52 
66 
65 

63 
60 
59 
65 

68 
54 
80 
68 

65 
57 
68 
66 

66 
60 
59 
67 

68 

Total  liquid 

assets 

53 
78 
73 

67 

57 
67 
70 

a  Defined  as  the  sum  of  deposits  in  Post  Office  Savings  Banks  and  Trustee  Savings  Banks, 
savings  certificates,  and  British  defense  bonds. 

"  Includes  deposits  in  other  savings  banks,  demand  deposits,  and  shares  and  deposits  in  build- 
ing societies  and  in  cooperative  societies. 

Source:  Adapted  from  M.  J.  Erritt  and  J.  L.  Nicholson,  "The  1955  Savings  Survey,"  Bul- 
letin of  the  Oxford  University  Institute  of  Statistics,  Vol.  20,   No.  2   (1958),  p.   127. 


the  survey  responses  by  independent  estimates  of  the  British  occupational 
distribution.  The  independent  estimates  of  aggregate  liquid  assets  are 
from  institutional  sources  and,  because  of  their  nature,  would  seem  to  be 
reasonably  accurate.  However,  it  is  not  clear  how  closely  they  approxi- 
mate individuals'  holdings  of  liquid  assets  because  the  estimates  include 
unknown  amounts  of  such  nonpersonal  holdings  as  trust  funds,  for  which 
there  was  apparently  no  reasonable  basis  for  estimating  the  amount.  On 
the  other  hand,  the  estimates  also  exclude  loan  stock  in  cooperative  soci- 
eties, which  many  respondents  apparently  included  among  their  liquid 
assets  and  which  could  not  be  segregated.22 

Unless  these  nonindividual  holdings  are  substantial,  it  is  clear  from 
Table  1  that  the  survey  estimates  greatly  understate  the  independent  esti- 
mates. In  the  case  of  "National  Savings"  (deposits  in  Post  Office  Savings 
Banks  and  Trustee  Savings  Banks,  savings  certificates,  and  British  defense 
bonds),  the  survey  accounts  for  barely  more  than  60  percent  of  the  insti- 
tutional aggregates.    For  other  liquid  assets    (deposits  in  other  savings 

22  M.  J.  Erritt  and  J.  L.  Nicholson,  "The  1955  Savings  Survey,"  Bulletin  of  the 
Oxford  University  Institute  of  Statistics,  Vol.  20,  No.  2   (May,  1958),  pp.  113-52. 
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banks,  demand  deposits,  and  shares  and  deposits  in  building  societies  and 
in  cooperative  societies),  the  survey  estimates  vary  from  47  percent  to 
77  percent  of  the  independent  estimate.  As  a  result,  the  survey  estimates 
of  total  liquid  assets  are  seen  to  understate  the  independent  aggregate  by 
roughly  half  to  one-third. 

Weighting  by  income  or  by  occupation  serves  to  increase  the  survey 
coverage  somewhat  more  than  does  the  use  of  unadjusted  data.  However, 
the  degree  of  understatement  is  substantial  and  "it  seems  clear  that  there 
is  considerable  under-reporting  of  assets,  even  though  we  do  not  know  the 
amount  of  nonpersonal  holdings  included  in  the  independent  estimates."23 

The  extent  to  which  the  survey  estimates  of  aggregate  saving  compare 
with  independent  estimates  is  shown  in  Table  2.  As  was  done  in  Table  1, 
three  sets  of  survey  percentages  are  shown  —  unadjusted  estimates,  esti- 
mates obtained  by  weighting  by  independent  estimates  of  the  income  dis- 
tribution, and  estimates  obtained  by  weighting  by  independent  estimates 
of  the  occupational  distribution. 

As  is  evident  from  this  table,  the  accuracy  of  the  survey  estimates 
seems  to  vary  substantially  with  the  type  of  saving.  The  survey  estimates 
of  contractual  saving  (life  insurance  premiums,  pension  contributions,  and 
repayment  of  mortgage  debt)  correspond  very  well  on  the  whole  with  the 
independently  estimated  aggregates.  However,  the  survey  estimates  of 
aggregate  saving  in  the  two  forms  of  liquid  assets  and  in  the  form  of 
"miscellaneous"  saving  (primarily  the  saving  of  unincorporated  business) 
invariably  understate  the  independently  estimated  amount  of  saving  or 
dissaving.  The  amount  of  saving  or  dissaving  in  "National  Savings"  is 
substantially  understated.  In  the  three  years  when  saving  in  this  form 
appears  to  have  been  negative,  the  actual  amount  of  dissaving  was  over- 
stated by  anywhere  from  150  to  350  percent,  and  in  the  one  year  when 
saving  was  positive,  the  survey  figures  indicated  dissaving. 

The  picture  is  even  worse  for  saving  in  the  form  of  other  liquid  assets. 
The  survey  estimates  of  this  form  of  saving  are  uniformly  negative, 
whereas  the  independent  estimates  point  to  positive  saving  during  all  four 
periods  studied.  Survey  estimates  of  "miscellaneous"  saving  also  bear  little 
relationship  to  the  independent  estimates.  As  often  as  not,  the  direction 
of  saving  indicated  by  the  survey  conflicts  with  the  direction  indicated  by 
the  independent  estimates.  In  this  instance,  however,  the  validity  of  the 
comparison  is  in  serious  doubt,  partly  because  the  correspondence  of  the 
survey  and  independent  concepts  of  business  saving  is  not  clear,  and  partly 
because  the  independent  estimates  were  obtained  as  residuals,  which  are 
invariably  subjected  to  appreciable  margins  of  error. 

It  is  interesting  to  note  that  with  a  few  exceptions  the  survey  estimates 

23  Ibi d.,  p.  129. 
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Table  2.   Comparison  of  Survey  Estimates  of  Aggregate 

Consumer  Saving  in  United  Kingdom  with 

Independent  Estimates,,  1952-55 


d 

Survey  estimate  as  a  percentage 

Period 

pendent 
estimate 

of  independent  estimate 

Type  of  saving 

(millions 

Un- 

Weighted 

Weighted 
by  occu- 
pation 

of  pounds) 

adjusted 

by  income 

Contractual11 

1951-52 

433 

100 

105 

105 

1952-53 

453 

90 

100 

97 

1953-54 

494 

101 

98 

93 

1954-55 

530 

95 

93 

98 

Change  in  "National 

Savings"1" 

1951-52 

-75 

352 

352 

333 

1952-53 

-97 

148 

139 

152 

1953-54 

-24 

300 

254 

262 

1954-55 

116 

-62 

-121 

-73 

Change  in  other 

liquid  assets'3 

1951-52 

104 

-162 

-171 

-194 

1952-53 

209 

-103 

-113 

-102 

1953-54 

233 

-82 

-104 

-79 

1954-55 

224 

-32 

-20 

-50 

Miscellaneous 

saving0 

1951-52 

-314 

-15 

-37d 

-13 

1952-53 

-335 

-7 

-13 

-18 

1953-54 

-403 

6 

-10 

15 

1954-55 

-617 

16 

15 

18 

Total  saving 

1951-52 

148 

32 

90d 

32 

1952-53 

230 

52 

64 

71 

1953-54 

300 

96 

94 

77 

1954-55 

260 

92 

98 

105 

a  Includes  life  insurance   premiums,   pension  contributions,   and  repayment  of  mortgage   debt. 

b  Defined  in  footnotes  to  Table  1. 

c  Primarily  unincorporated  business  saving,  but  includes  also  net  purchase  of  land,  real  estate, 
securities  and  annuities,  property  improvement,  gifts  and  loans,  and  reduction  of  outstanding  debt. 

d  Excludes  adjustment  of  business  saving  for  underestimate  of  business  expenses  in  1952  survey 
because  of  incomplete  coverage  of  the  relevant  questions  in  that  year. 

Source:  Adapted  from  M.  J.  Erritt  and  J.  L.  Nicholson,  "The  1955  Savings  Survey,"  Bul- 
letin of  the  Oxford  University  Institute  of  Statistics,  Vol.  20,  No.  2  (1958),  pp.  129-40. 


obtained  by  weighting  by  income  or  by  occupation  are  no  better  than  the 
unadjusted  estimates.  Also  pertinent  is  the  fact  that  experimentation  with 
alternative  means  of  estimating  saving  did  not  yield  any  better  estimates 
than  those  shown  in  Table  2.  The  survey  estimates  in  that  table  were 
based  on  the  change  in  holdings  as  reported  in  the  last  survey  in  the  given 
period.24  Since  respondents  were  also  asked  for  holdings  in  each  of  these 
four  surveys,  saving  could  also  be  estimated  as  the  change  in  holdings 
reported  on  two  different  dates,  or,  by  hindsight,  as  the  change  in  the 
previous  year's  holdings  of  respondents  interviewed  the  year  before.   Only 

24  For  example,  the  estimate  of  1951-52  saving  was  based  on  respondent  re- 
ports of  saving  in  the  survey  conducted  in  April,  1952. 
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the  first  of  these  alternatives  appears  to  have  been  tested,  and  it  produced 
results  no  better  than  those  shown  in  Table  2.  From  data  in  the  original 
study,25  it  is  hardly  likely  that  the  other  two  alternatives  would  have 
proved  superior,  because  few  substantial  differences  were  apparent  be- 
tween current  holdings  reported  one  year  and  current  holdings  reported 
the  following  year.  Clearly,  failure  to  obtain  complete  and  accurate  data 
was  a  serious  shortcoming  of  these  surveys. 

Savings  Surveys  in  the  United  States 

At  the  present  time,  three  survey  operations  seek  data  on  the  saving  of 
American  consumers  on  a  nationwide  basis.  Two  are  conducted  only 
occasionally,  whereas  the  other,  although  conducted  annually,  does  not 
cover  all  types  of  assets  and  liabilities.  The  former  are  the  expenditure- 
income  studies  of  the  United  States  Bureau  of  Labor  Statistics  and  the 
survey  of  family  financial  characteristics  of  the  Federal  Reserve  Board. 
Although  conducted  primarily  to  compile  data  to  serve  as  weights  for  the 
cost  of  living  index,  the  BLS  surveys  in  recent  years  have  also  been  seeking 
estimates  of  family  saving  by  acquiring  data  on  individual  assets  and 
liabilities.  However,  since  these  surveys  are  conducted  only  every  10  to 
15  years,  available  data  on  consumer  saving  from  this  source  are  as  yet 
very  sparse.  The  Federal  Reserve  survey  has  been  conducted  in  1963  and 
1964  and  covers  all  assets  and  debts. 

The  third  survey,  the  Survey  of  Consumer  Finances,  is  conducted  every 
year  by  the  Survey  Research  Center,  but  only  savings  in  selected  assets  and 
liabilities  are  covered  each  time.  In  addition,  the  coverage  of  different 
assets  and  liabilities  has  varied  from  year  to  year. 

The  two  earlier  surveys  have  been  utilized  at  various  times  as  a  basis 
for  estimating  aggregate  consumer  saving  in  different  forms;  the  Federal 
Reserve  survey  data  are  only  beginning  to  be  used  for  this  purpose  and  no 
results  are  as  yet  available.  The  following  pages  review  the  results  of 
these  surveys. 

Expenditure-Income  Studies 

The  first  attempt  to  collect  complete  data  on  saving  and  debt  by  means 
of  a  nationwide  consumer  survey  appears  to  have  been  made  as  part  of 
the  1935-36  Consumer  Purchases  Study.  The  interviews  on  this  survey 
sought  not  only  a  complete  breakdown  of  income  and  expenditures,  but 
also  data  on  changes  in  each  of  the  assets  and  liabilities  held  by  the  fam- 
ily. However,  because  of  the  scope  of  the  study  and  its  emphasis  on  in- 
come and  expenditures,  the  questions  on  saving  were  accorded  secondary 

25  Erritt  and  Nicholson,  op.  cit.,  p.  126. 
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importance,  and  indications  are  that  they  may  have  been  treated  in  this 
manner  during  the  course  of  the  field  work.  In  addition,  the  coverage  of 
the  sample  was  limited  primarily  to  native-born  husband-wife  families. 
Furthermore,  the  representation,  and  apparently  the  responses,  at  the 
upper-income  levels  was  so  poor  that  these  data  were  not  considered  reli- 
able enough  to  serve  as  a  basis  for  aggregation.  Hence,  saving  aggregates 
were  obtained  by  combining  the  survey  responses  for  families  earning 
under  $10,000  a  year  with  estimates  for  (1)  families  in  these  income  levels 
who  had  been  excluded  from  the  survey  and  for  (2)  families  earning  over 
$10,000  a  year  in  1935-36.26  The  resulting  estimates,  therefore,  represent 
only  in  part  —  and  for  many  categories  of  saving,  only  a  very  small  part  — 
the  responses  obtained  in  the  1935-36  Consumer  Purchases  Study. 

Perhaps  for  this  reason,  the  survey  aggregates  for  1935-36  do  not  as  a 
rule  depart  too  far  from  the  national  income  figures,  as  is  shown  in 
Table  3.  Although  comparisons  between  survey  aggregates  and  external 
data  are  necessarily  very  rough,  the  comparison  for  1935-36  is  even  more 
so  because  of  the  many  assumptions  that  had  to  be  made  regarding  the 
saving  of  that  part  of  the  population  not  covered  in  the  survey.  These 
various  sets  of  assumptions  led  to  eight  different  estimates  for  each  aggre- 
gate.27 Since  the  relative  superiority  of  these  different  estimates  is  an 
unknown  question,  two  accuracy  percentages  for  each  form  of  saving  are 
shown  in  Table  3,  covering  the  entire  range  of  the  survey  estimates.  This 
range  is  in  many  cases  fairly  large,  and  as  a  result  definitive  judgments  on 
the  accuracy  of  these  estimates  are  not  easy  to  make. 

It  is  also  difficult  to  ascertain  the  extent  to  which  particular  deviations 
may  be  due  to  the  poor  quality  of  the  survey  data  and  the  extent  to  which 
they  may  be  due  to  faulty  assumptions  regarding  the  saving  of  that  part 
of  the  population  not  covered  by  the  survey.  Indeed,  comparison  of  the 
percentages  in  Table  3  for  1935-36  with  the  accuracy  percentages  for  1941 
would  seem  to  indicate  that  the  estimation  procedures  for  the  excluded 
parts  of  the  population  may  have  been  primarily  responsible  for  the  rela- 
tively good  comparisons  obtained  by  the  1935-36  survey  aggregates. 

For  this  reason,  the  1941  estimates  probably  provide  a  better  indica- 
tion of  the  quality  of  the  savings  data  collected  in  these  prewar  surveys. 
These  aggregates,  which  are  based  entirely  on  the  nationwide  study  of 
family  spending  and  saving  in  1941-42,  indicate  that  on  the  whole  saving 
in  the  form  of  assets  was  underestimated  but  that  dissaving  (accumulation 

28  Dorothy  S.  Brady,  "Family  Saving,  1888-1950"  in  R.  W.  Goldsmith,  D.  S. 
Brady,  and  H.  Mendershausen,  A  Study  of  Saving  in  the  United  States,  Vol.  3 
(Princeton:    Princeton  University  Press,  1956),  p.  158-78. 

27  Ibid.,  Table  H-13.  There  are  two  alternative  estimates  of  the  saving  pattern 
of  excluded  relief  and  farm  families  and  four  alternative  estimates  of  the  saving 
and  saving  distribution  of  those  earning  over  $10,000. 
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Table  3.  Comparison  of  1935-36  and  1941  Estimates  of  Aggregate 
Saving  and  Dissaving  with  National  Income  Estimates 


Item 


Survey  estimate  as  a 

percentage  of  national 

income  figure 


1935-36a 


1941 


Currency  on  hand 

Bank  accounts 

Checking  accounts 

Savings  accounts 

Investment  in  business 

Nonfarm 

Farm  business  and  real  estate .  . 
Investment  in  nonfarm  real  estate. 
Securities 

United  States  Government 

Other 

Insurance  premiums  paid 

Insurance  premiums  surrendered . 

Insurance  policies  settled 

Mortgage  loans  lent 

Social  Security  contributions 

Old  age  survivors  contributions. 

Unemployment  insurance 


Total  of  matched  asset  items , 


Mortgage  debt 

Nonfarm  debt  to  banks  and  insurance  companies. 

Nonfarm  charge  accounts 

Nonfarm  installment  balances 

Other  nonfarm  debt 


Total  of  liability  items . 


66  to  150 

87  to  110 

134  to  191 

59  to  72 

115  to  135 

-84  to  94 

■218to-369d 

33  to  79 

-57  to  -104d 

n.a. 

n.a. 

103  to  122 

n.a. 

n.a. 

n.a. 

n.a. 

n.a. 

n.a. 


135  to  159 


116  to  91° 

114  to  176 

111  to  162 

86  to  98 

440  to  606 


85  to  128 


27 

-50b 

315° 

34 

102 

99 

105 

88 

43 

42 

38c 

78 

15 

-14b 

-61b 

78 

78 

75 


61 


92 

495 
130 
305 
448 


168 


tt  1935-36  comparisons  based  on  two  alternative  estimates  regarding  saving  of  farm  families  not 
covered  in  the  1935-36  Consumer  Purchases  Study.  The  assumptions  underlying  the  estimates  were 
that  the  saving  of  these  families  (which  included  families  on  relief  as  well  as  nonrelief  families 
which  had  neither  a  husband-wife  present  or  were  nonwhite)  was,  in  turn,  zero,  and  the  same  as 
that  of  eligible  families  at  the  same  income  level. 

b  Survey  estimate  negative,  national  income  estimate  positive. 

c  Both  estimates  negative. 

d  Survey  estimate  positive,  national  income  estimate  negative. 

Source:  Adapted  from  Tables  H-19  and  H-20  of  Dorothy  S.  Brady,  "Family  Saving,  1886- 
1950,"  in  R.  W.  Goldsmith,  D.  S.  Brady,  and  H.  Mendershausen,  A  Study  of  Saving  in  the  United 
States  (Princeton:    Princeton  University  Press,  1956),  pp.  174-78. 


of  debt)  was  overestimated.  Particularly  striking  is  the  very  poor  coverage 
of  saving  in  the  forms  of  currency,  bank  accounts,  securities,  and  mort- 
gage loans  lent.  In  the  latter  case,  the  survey  aggregate  indicated  dissav- 
ing, whereas  the  national  income  aggregate  indicated  positive  saving.  In 
the  case  of  currency  and  savings  accounts,  only  about  one-third  of  the 
total  saving  in  these  forms  could  be  accounted  for  by  the  survey. 

On  the  other  hand,  dissaving  in  the  form  of  personal  loans,  installment 
credit,  and  other  nonfarm  debt  was  substantially  overestimated.  The  devi- 
ations are  so  large,  being  of  the  order  of  300  percent  to  500  percent,  as  to 
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Table  4.  Comparison  of  1950  BLS- Wharton  Estimates 

of  Aggregate  Urban  Saving  with  SEC  Aggregates 

for  Total  United  States  Saving 

(billions  of  dollars) 


Item 


BLS-Wharton 

estimate,  1950 

urban 

United  States 


Adjusted  SEC 

aggregate,  total 

United  States 


Cash  and  deposits 

Securities 

Net  business  investment  (before  inventory 

valuation  adjustment) 

Private  insurance  contributions 

Life  insurance  premiums 

Change  in  equity  in  nonfarm  property. . .  . 

Improvement  of  nonfarm  property 

Nonfarm  mortgages 

Other  consumer  debt 


3.9 
.  4  to  .8 


Total  saving,  SEC  concept" 


5.9 


11.3  to  11.7 


a  Sums  of  individual  figures  will  not  add  to  total  saving  because  some  items  were  not  included 
in  the  SEC  concept  and  others  are  not  shown  because  comparable  data  were  not  available  from 
both  sources. 

Source:  Adapted  from  Irwin  Friend  and  Stanley  Schor,  Who  Saves?  published  as  a  supple- 
ment to  the  Review  of  Economics  and  Statistics,  Vol.  41,  No.  2   (May,   1959),  Table  4. 


indicate  that  either  the  respondents  misunderstood  the  questions  or  that 
the  survey  was  not  obtaining  information  comparable  with  the  national 
income  data.28 

The  savings  estimates  obtained  from  the  1950  Survey  of  Consumer 
Expenditures  appear  to  have  been  more  accurate  in  many  respects,  al- 
though comparison  is  difficult  because  the  coverage  of  that  survey  was 
restricted  to  urban  areas,  whereas  the  correpsonding  data  on  aggregate 
saving  refer  to  both  rural  and  urban  areas.  As  a  result,  the  survey  esti- 
mates of  saving  in  1950,  shown  in  Table  4,  would  be  expected  to  fall 
considerably  below  the  corresponding  figures  for  the  total  nation  for  most 
assets.  If  the  urban-rural  income  distribution  were  any  indication  of  the 
urban-rural  saving  distribution,  the  1950  survey  should  have  accounted 
for  about  73  percent  of  individuals'  saving  in  that  year.29  However,  since 
the  distribution  of  wealth  is  generally  more  concentrated  than  the  distri- 
bution of  income,  this  percentage  is  undoubtedly  a  lower  limit  to  the 

28  It  has  been  suggested  that  much  of  this  discrepancy  may  have  resulted  from 
misplaced  income  in  the  reporting  process  (ibid.,  p.  178).  Examination  of  the 
original  questionnaire  form,  however,  raises  doubt  regarding  the  validity  of  this 
explanation. 

=I)  This  percentage  is  given  in  the  monograph  by  Irwin  Friend  and  Stanley 
Schor,  Who  Saves?  published  as  a  supplement  to  the  Review  of  Economics  and 
Statistics,  Vol.  41,  No.  2  (May,  1959),  Table  4. 
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actual  proportion  of  the  nation's  saving  accounted  for  by  the  urban  areas 
in  1950.  It  is  also  clear  that  saving  in  certain  types  of  assets,  such  as  in 
securities,  is  almost  entirely  accounted  for  by  the  urban  population. 

The  comparisons  in  Table  4  indicate  that  for  some  types  of  saving  the 
1950  survey  data  appear  to  be  highly  reliable.  This  is  the  case  for  saving 
in  the  form  of  life  insurance,  change  in  equity  in  nonfarm  property,  im- 
provement of  nonfarm  property,  and  nonmortgage  debt.  For  other  forms 
of  saving  not  much  can  be  said  about  the  accuracy  of  the  survey  data, 
either  because  the  concepts  underlying  the  particular  form  of  saving  may 
not  be  the  same  in  both  cases  (as  is  true  of  net  business  investment)  or 
because  considerable  uncertainty  surrounds  the  accuracy  of  the  external 
aggregate  estimate  (as  in  the  case  of  securities)  .30 

In  estimating  the  amount  of  saving  in  the  form  of  cash  and  bank 
deposits,  the  1950  survey  estimates  differ  completely  from  the  apparent 
actual  figures.  Not  only  is  the  discrepancy  huge,  being  of  the  order  of 
$7.5  billion,  but  the  direction  of  change  is  completely  missed  (the  survey 
indicating  substantial  dissaving,  whereas  other  sources  point  to  appreci- 
able saving  in  these  forms  during  1950).  Here,  as  was  the  case  with  so 
many  previous  surveys,  the  data  appear  to  have  been  clearly  inadequate. 
Since  only  one  question  was  asked  on  all  of  these  forms  of  saving  —  cash 
on  hand;  checking  accounts;  savings  accounts  in  commercial  and  savings 
banks;  and  savings  and  other  accounts  in  savings  and  loan  associations,  in 
building  and  loan  associations,  in  credit  unions,  and  in  postal  savings  — 
it  is  not  possible  to  determine  the  extent,  if  any,  to  which  this  discrepancy 
varied  with  these  different  types  of  saving.  Whatever  the  situation  in  this 
regard,  the  principal  cause  may  well  have  been  the  estimation  of  saving 
as  the  difference  between  current  holdings  and  holdings  as  of  a  year 
earlier,  accentuated  by  placing  these  questions  toward  the  end  of  an  inter- 
view that  lasted  anywhere  from  four  to  eight  hours.  If  the  respondent 
had  much  reluctance  initially  at  divulging  information  on  his  assets,  this 
reluctance  undoubtedly  would  have  been  increased  by  the  time  these 
questions  were  asked  and  would  have  manifested  itself  primarily  in  under- 
reporting of  current  holdings  rather  than  underreporting  of  past  holdings. 

As  a  result  of  this  understatement  of  saving  in  the  form  of  cash  and 
deposits,  the  survey  estimate  of  total  urban  saving  in  1950  comes  to  little 
more  than  half  of  the  SEC  estimate  of  the  saving  of  all  individuals  in  that 

30  The  SEC  estimate  in  the  form  of  securities  depends  heavily  on  whether  or 
not  brokerage  commissions  paid  on  purchases  and  sales  of  outstanding  issues  are 
excluded  from  reported  net  purchases  of  securities.  As  noted  by  Friend  and  Schor 
{ibid.,  pp.  85-86),  the  survey  questionnaires  do  not  indicate  what  course  was 
followed.  Friend  and  Schor  chose  to  exclude  these  commissions  from  the  SEC 
estimate  of  saving;  if  the  figure  had  been  included,  the  SEC  estimate  would  have 
been  doubled,  as  shown  in  Table  4. 
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year.  Adjusted  for  the  exclusion  of  the  rural  areas,  the  survey  estimate 
rises  to  $8.1  billion  as  compared  with  the  SEC  figure  of  $11.3  billion.  No 
appreciable  improvement  in  the  survey  estimate  is  obtained  if  various 
regressions  between  saving  and  income  are  used  to  obtain  an  aggregate 
figure.31 

Survey  of  Consumer  Finances 

Contrary  to  the  procedure  in  the  consumer  expenditure  studies,  infor- 
mation on  asset  holdings  has  constituted  a  central  portion  of  the  annual 
interviews  of  the  Survey  of  Consumer  Finances.  The  general  approach 
has  been  to  ask  first  a  number  of  questions  on  asset  holdings,  or  on  saving, 
as  well  as  on  income  of  the  spending  unit.  Questions  on  debt  were  asked 
at  times  as  part  of  the  questions  on  assets  and  at  other  times  to  a  large 
extent  in  connection  with  the  questions  on  recent  purchases  of  durable 
goods. 

The  coverage  of  the  assets  has  varied  appreciably  since  these  annual 
surveys  began.  Nevertheless,  almost  all  the  annual  surveys  have  asked  for 
information  on  liquid  assets,  that  is,  on  holdings  in  checking  accounts, 
savings  accounts,  and  in  United  States  government  bonds.  In  addition, 
questions  have  been  asked  occasionally  on  holdings  of  corporate  stock  and, 
on  the  debt  side,  on  the  amount  outstanding  by  type  of  debt.  As  a  result 
of  these  questions,  and  of  the  availability  of  roughly  corresponding  aggre- 
gates from  the  SEC  estimates  of  individuals'  savings  as  well  as  from  the 
more  recent  Federal  Reserve  flow-of-funds  data,  a  fair  amount  of  infor- 
mation is  available  for  evaluating  the  reliability  of  these  figures. 

Comparison  of  selected  asset  and  debt  holdings  from  the  Survey  of 
Consumer  Finances  of  1958  with  corresponding  estimates  from  the  Fed- 
eral Reserve  flow-of-funds  system  is  shown  in  Table  5.  As  is  evident  from 
this  table,  the  pattern  of  correspondence  between  the  survey  estimates  and 
the  external  estimates  is  fairly  similar  to  that  of  the  BLS  consumer  expen- 
diture surveys  and  to  that  of  the  British  surveys  discussed  earlier.  Hold- 
ings of  liquid  assets,  excluding  cash,  are  understated  substantially,  partic- 
ularly holdings  of  government  bonds  and  of  savings  accounts.  In  each 
of  these  cases,  less  than  half  of  the  estimated  amount  in  the  hands  of 
individuals  was  accounted  for  by  the  survey.  The  correspondence  is 
somewhat  better  in  the  case  of  checking  accounts,  but  even  for  this  typi- 
cally less-sensitive  asset,  the  survey  aggregate  is  about  30  percent  below  the 
estimated  national  total. 

Although  the  flow-of-funds  estimates  of  these  aggregates  are  undoubt- 
edly subject  to  errors,  these  errors  are  hardly  likely  to  explain  such  sub- 

31  Friend  and  Schor,  ibid.,  pp.  22-23. 
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Table  5.  Comparison  of  1958  Survey  of  Consumer  Finances 

Estimates  of  Holdings  of  Selected  Assets  and  Debts 

with  Flow-of-Funds  Estimates  for  End  of  1957, 

Total  United  States 


Category 

Flow-of-funds 

estimate 

(billions  of  dollars) 

SCF  estimate  as 

a  percentage  of 

flow-of-funds 

estimate 

United  States  Government  bonds 

65 

32 

129 

226 

295a 

101 

34 

15 

149 

42 

Checking  accounts 

69 

46 

48 

26a 

97 

Nonmortgage  installment  debt 

56 

Nonmortgage  noninstallment  debt 

Total  comparable  debt 

33b 
81b 

a  Early  1957  estimates. 

b  This  does  not  allow  for  the  fact  that  charge  accounts  are  excluded  from  the  SCF  coverage 
but  are  included  in  the  flow-of-funds  estimates. 

Source:    Federal  Reserve  Bulletin,  Vol.  44,  No.  9  (September,  1958),  pp.  1047-50. 


stantial  differences.32  A  major  source  of  error  appears  to  lie  in  the  inade- 
quate reporting  of  holdings  by  the  survey  respondents.  Thus,  according 
to  a  survey  of  demand  deposit  ownership  conducted  among  financial  in- 
stitutions by  the  Federal  Home  Loan  Bank  Board,  about  45  million  check- 
ing accounts  were  owned  by  individuals  and  farm  operators  in  early  1958, 
whereas  the  estimate  of  this  number  obtained  from  the  Survey  of  Con- 
sumer Finances  is  33  million.33  On  the  basis  of  another  survey  conducted 
by  the  same  agency  among  financial  institutions,  individuals  and  farm 
operators  were  estimated  to  own  22  million  savings  and  loan  share  ac- 
counts at  the  end  of  1957;  the  estimate  obtained  from  the  Survey  of 
Consumer  Finances  for  early  1958  is  8  million.34  Thus  there  is  little  doubt 
that,  as  in  the  other  surveys,  many  holdings  of  these  assets  are  not  re- 
ported, and  probably  other  holdings  are  understated. 

The  substantial  understatement  of  holdings  of  corporate  stock  may  well 
be  due  to  an  even  stronger  manifestation  of  the  same  phenomenon.  In 
addition,  a  large  part  of  this  discrepancy  is  undoubtedly  due  to  the  much 
greater  degree  of  concentration  of  these  holdings  than  of  liquid  assets. 

32  For  a  discussion  of  the  possible  sources  of  error  in  the  flow-of-funds  estimates 
of  these  aggregates,  see  the  Technical  Appendix  to  "The  Financial  Position  of 
Consumers"  in  the  Federal  Reserve  Bulletin,  Vol.  44,  No.  9  (September,  1958), 
pp.  1047-51. 

33  Ibid.,  p.  1049. 

34  The  actual  discrepancy  is  somewhat  less  than  is  indicated  to  the  extent  that 
dead  and  forgotten  accounts  exist  but  cannot  be  picked  up  on  a  consumer  survey. 
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Figure  1.    Estimated  Consumer  Holdings  of  Selected  Liquid 

Assets  From  Survey  of  Consumer  Finances  and 

From  Flow-of-Funds,  1949-58a 

BILLIONS  of  dollars 
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*  Flow-of-funds  estimates  are  for  end  of  years  1948-57  inclusive;  survey  figures  are  for 
early  in  years,  1949-58,  inclusive. 

Source:     Federal  Reserve  Bulletin,  Vol.  44,  No.  9   (September,   1958),  p.   1047. 


As  has  been  noted  by  the  Federal  Reserve  in  an  evaluation  of  this  dis- 
crepancy, 

The  chance  inclusion  of  a  single  large  holding  can  have  a  tremendous  impact  on 
the  estimated  aggregate  on  consumer  holdings.  For  example,  one  respondent  in 
the  1953  survey  reported  owning  about  $1  million  in  corporate  stock.  His  case 
alone  raised  the  estimated  aggregate  from  $37  billion  to  $53  billion. 

No  doubt,  part  of  the  difference  can  be  explained  by  this  factor  and 
other  parts  by  the  fact  that  the  flow-of-funds  estimates  include  holdings 
of  personal  trusts  and  of  stock  not  publicly  traded.  The  fact  remains, 
however,  that  the  survey  estimates  have  consistently  understated  external 
estimates  of  consumer  holdings  of  corporate  stock  as  well  as  of  liquid  assets. 
If  anything,  the  discrepancy  rose  between  1949  and  1958,  as  is  shown  in 
Figure  1.  Over  this  period,  the  survey  estimates  have  been  roughly  half 
the  size  of  the  flow-of-funds  estimates  and,  with  the  base  increasing  over 
time,  the  dollar  amount  of  discrepancy  has  tended  to  rise. 

In  the  case  of  consumer  debt,  the  survey  estimates  correspond  very  well 
in  some  instances  with  the  flow-of-funds  data.  Thus,  the  survey  estimate 
of  nonfarm  mortgage  debt  is  very  close  to  the  flow-of-funds  estimate  based 
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on  lender  reports,  as  has  apparently  been  the  case  in  previous  years  also.35 
On  the  other  hand,  the  survey  estimate  of  nonmortgage  debt  is  substan- 
tially lower  than  the  flow-of-funds  estimate.  Both  installment  and  non- 
installment  debt  appear  to  be  understated  substantially,  and  this  seems  to 
have  been  the  situation  in  previous  years  as  well.36  As  with  liquid  assets, 
the  evidence  points  to  a  widespread  underreporting  of  debt  as  the  prin- 
cipal cause  of  this  discrepancy. 

Turning  to  estimates  of  saving,  the  Survey  of  Consumer  Finances  pre- 
sents a  somewhat  mixed  picture.  Total  consumer  saving  as  estimated 
from  the  Survey  of  Consumer  Finances  —  usually  as  the  change  between 
holdings  reported  currently  and  holdings  reported  as  of  the  previous  year 
—  agrees  well  with  the  aggregate  obtained  from  other  sources.  However, 
this  is  because  the  substantial  understatement  of  saving  in  the  form  of 
liquid  assets  is  counterbalanced  by  a  substantial  overstatement  in  the  esti- 
mate of  saving  in  the  form  of  nonliquid  assets.  Thus,  the  picture  for  the 
years  1947-50  is  as  follows: 37 

SCF  SEC 

(billions  of  dollars) 

Saving  in  liquid  assets —22.5  7.9 

Saving  in  nonliquid  assets 71 .3  39.4 

Total  saving 48 . 8  47 . 3 

A  more  detailed  comparison  of  these  two  sets  of  data  is  shown  for  1950 
in  Table  6.  Here  again,  perhaps  the  most  striking  feature  is  the  cor- 
respondence of  the  two  aggregate  estimates  of  total  saving,  despite  tre- 
mendous differences  in  the  estimates  of  saving  in  particular  forms.  Most 
striking  of  all  is  the  difference  of  almost  $11  billion  between  the  two  esti- 
mates of  aggregate  saving  in  the  form  of  cash  and  deposits,  a  difference 
almost  as  large  as  estimated  total  saving  that  year.  Nevertheless,  this  dis- 
crepancy is  offset  by  substantial  discrepancies  in  the  opposite  direction  in 
the  estimates  of  saving  in  securities,  in  nonmortgage  debt,  and  particularly 
in  net  business.38  It  is  apparently  because  of  such  discrepancies  that  the 
Survey  of  Consumer  Finances  has  found  little  use  in  the  preparation  of 

30  Technical  Appendix  to  "The  Financial  Position  of  Consumers,"  op.  cit., 
pp.  1050-51. 

36  Federal  Reserve  Bulletin,  Vol.  42,  No.  7   (July,  1956),  pp.  696-701. 

37  Reports  of  Federal  Reserve  Consultant  Committees  on  Economic  Statistics, 
Hearings  Before  the  Subcommittee  on  Economic  Statistics  of  the  Joint  Committee 
on  the  Economic  Report,  84th  Congress,  1st  Session,  July  19-26  and  October  4 
and  5,  1955,  p.  282. 

33  It  is  interesting  to  note  that  these  discrepancies  would  have  been  increased 
even  further  had  the  estimates  been  obtained  by  computing  regressions  of  saving 
on  income  or  by  using  external  estimates  of  the  consumer  income  distribution;  see 
Friend  and  Schor,  op.  cit.,  pp.  28-30. 
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Table  6.   Comparison  of  1950  Survey  of  Consumer  Finances 

Estimates  of  Aggregate  Saving  with  Corresponding  SEC 

Estimates  of  Total  United  States  Saving 

(billions  of  dollars) 


Item 

Survey  of  Con- 
sumer Finances 
estimate 

Adjusted  SEC 
estimate 

Cash  and  deposits 

All  securities 

United  States  government  securities 

Net  business  investment  (before  inventory 

valuation  adjustment) 

Life  insurance  premiums 

-6.9 
3.1a 
1.1 

5.1 

5.5 

-2.0 

12.6 

3.9 
.4a 
-.6 

0 

5.7 

-4.2 

Total  saving,  SEC  conceptb 

11.3 

a  Urban  United  States,  estimate  for  all  securities  from  p.  27  of  this  publication. 

b  Sums  of  individual  figures  will  not  add  to  total  saving  because  some  items  are  not  included 
in  the  SEC  concept  and  others  are  not  shown  because  comparable  data  were  not  available  from 
both  sources. 

Source:  Adapted  from  Irwin  Friend  and  Stanley  Schor,  Who  Saves?  published  as  a  supple- 
ment to  the  Review  of  Economics  and  Statistics,  Vol.  41,  No.  2  (May,  1959),  Table  4. 


national  income  estimates  or  in  the  fiow-of-funds  accounts,39  although 
they  have  been  used  widely  in  studies  of  consumer  behavior. 


Overall  Comments 

In  virtually  none  of  the  studies  mentioned  in  this  chapter  are  aggregate 
estimates  of  financial  data  available  from  external  sources  with  known 
(small)  margins  of  error  and  on  a  basis  directly  comparable  with  the  survey 
data.  In  most  instances,  these  external  estimates  come  either  from  institu- 
tional sources  or  from  national  income  studies,  which  are  themselves  based 
primarily  on  institutional  reports.  As  a  result,  only  rough  adjustment  can 
be  made  for  conceptual  differences  in  the  two  sources,  and  unknown  por- 
tions of  the  observed  discrepancies  are  attributable  either  to  the  adjust- 
ments that  are  made  or  to  adjustments  that  should  have  been  made  but  for 
which  no  basis  existed  for  doing  so.  Despite  these  differences,  the  dis- 
crepancies in  many  cases  are  so  substantial  and  consistent  from  one  survey 
to  another  that  there  is  little  doubt  regarding  the  inadequacy  of  the  past 
survey  data  in  various  respects. 

For  one  thing,  survey  aggregates  tend  to  understate  the  apparent  true 
figures  in  estimating  most  financial  magnitudes.  The  underestimates  ap- 
pear to  be  relatively  low  for  total  saving,  generally  of  the  order  of  10  to 

39  Arthur  L.  Broida,  "Consumer  Surveys  as  a  Source  of  Information  for  Social 
Accounting  —  The  Problems,"  paper  delivered  at  the  Conference  on  Research  on 
Income  and  Wealth,  1959,  and  published  in  Studies  in  Income  and  Wealth,  Vol. 
26,  National  Bureau  of  Economic  Research  (Princeton:  Princeton  University  Press, 
1962). 
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20  percent.  Estimates  of  total  asset  holdings  appear  to  be  understated 
much  more,  although  the  extent  of  understatement  varies  tremendously 
with  the  particular  asset.  This  understatement  appears  to  be  largest  for 
the  income-earning  assets  and  for  income  derived  therefrom,  particularly 
for  the  more  liquid  assets.  Thus,  in  the  case  of  income,  the  largest  dis- 
crepancies percentagewise  are  encountered  in  reports  of  interest  and  divi- 
dend receipts,  whereas  with  asset  holdings  the  largest  discrepancies  appear 
in  reports  of  holdings  of  savings  accounts,  government  bonds,  and  corpo- 
rate stock,  with  saving  in  these  forms  also  understated  substantially. 

Although  liquid  assets  and  the  income  earned  therefrom  are  under- 
stated, many  nonliquid  assets  and  saving  in  these  forms  appear  to  be  over- 
stated, for  reasons  not  clear  from  the  aggregative  comparisons.  Thus, 
saving  in  the  form  of  investment  in  own  business,  in  nonfarm  real  estate, 
in  farm  real  estate,  and  in  improvements  on  property  tend,  if  anything,  to 
exceed  the  corresponding  estimates  of  these  magnitudes  from  the  national 
income  data.  Since  national  income  estimates  of  these  magnitudes  are  sub- 
ject to  considerable  margins  of  error,  the  principal  fault  may  not  lie  with 
the  survey  data.  Certainly,  at  least  part  of  the  discrepancy  seems  to  be  at- 
tributable to  the  different  ways  in  which  investment  in  own  business  has 
been  defined.  Nevertheless,  it  would  be  surprising  if  there  were  not  also 
errors  in  the  survey  data,  particularly  since  the  poorest  cooperation  is 
generally  obtained  from  the  entrepreneurial  class. 

Although  failure  to  report  and  errors  in  data  that  are  reported  appear 
to  permeate  information  from  all  economic  groups,  these  errors  appear  to 
be  more  pronounced  among  surveys  of  the  wealthier  class.  At  the  same 
time,  the  Federal  Reserve  estimates  suggest  that  holdings  of  the  more 
liquid  assets  are  in  many  cases  not  being  reported,  and  these  instances  are 
much  too  frequent  to  be  accounted  for  only  by  the  upper-income  groups. 

Although  the  aggregative  comparisons  cannot  provide  conclusive  evi- 
dence on  the  causes  of  survey  biases  or  on  the  relative  effectiveness  of 
different  survey  approaches,  the  data  presented  in  this  chapter  do  provide 
a  number  of  important  inferences  regarding  the  value  of  different  survey 
techniques.  Since  these  inferences  are  to  be  tested  more  thoroughly  in 
later  chapters,  it  is  of  particular  interest  at  this  point  to  list  them  briefly. 

One  point  on  which  the  aggregative  comparisons  provide  reasonably 
clear  evidence  is  on  the  highly  skewed  distribution  of  assets  and  of  savings. 
As  a  result,  it  appears  that  more  reliable  data  on  consumer  financial  be- 
havior are  obtainable  if  heavy  oversampling  of  the  upper-income  groups 
can  be  incorporated  into  a  survey  design,  as  is  the  case  with  the  Survey  of 
Consumer  Finances.  Second,  it  is  clear  that  relatively  poor  cooperation 
is  obtained  from  the  high-income  groups,  which  tends  to  reduce  the  effec- 
tive representation  from  these  income  levels  at  the  same  time  that  the 
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reports  which  are  obtained  possess  a  greater  likelihood  of  containing  sub- 
stantial errors.  This,  then,  is  yet  another  basis  for  disproportionately  high 
representation  from  these  groups.  For  similar  reasons,  heavy  concentra- 
tion is  needed  on  the  entrepreneurial  sector;  in  this  case,  not  only  is  the 
problem  of  poor  cooperation  encountered,  but  both  the  meaning  of  the 
questions  and  the  underlying  concepts  are  easily  misinterpreted. 

A  basis  for  optimism  exists  in  the  growing  recognition  of  these  prob- 
lems, which  led  in  more  recent  years  to  studies  designed  specifically  to 
explore  means  of  obtaining  financial  data  from  the  very  wealthy40  and  to 
much  higher  disproportionate  representation  in  savings  surveys  from  the 
upper-income  groups,  especially  so  in  the  recent  surveys  of  consumer 
finances  and  in  the  Federal  Reserve  surveys.  Such  techniques  promise 
to  improve  considerably  the  coverage  of  financial  assets  and  also  serve  to 
throw  more  light  on  the  reliability  of  the  older  studies. 

The  aggregate  comparisons  also  suggest  that  in  many  surveys  the 
wrong  member  of  the  consumer  unit  may  be  interviewed.  The  respondent 
on  most  of  the  past  surveys  has  been  the  housewife,  who  in  many  cases  is 
not  well  acquainted  with  the  family's  finances.  This  bias  tends  to  be  rein- 
forced by  the  broad  coverage  of  the  questions,  which  have  often  related 
to  all  members  of  the  family  or  consumer  unit.  Even  if  the  respondent  is 
familiar  with  his  family's  finances,  he  may  not  know  how  various  family 
members  are  handling  their  savings,  or  how  much  they  have,  particularly 
other  working  members. 

Placing  the  primary  emphasis  on  subjects  other  than  family  finances  is 
another  danger  in  savings  surveys.  In  such  cases,  the  interviews  tend  to 
be  so  long  and  involved  that  by  the  time  the  subject  of  finances  is  reached 
both  the  interviewer  and  the  respondent  are  fatigued,  with  the  result  that 
both  are  anxious  to  run  through  the  questions  rather  than  to  concentrate 
on  accuracy  or  completeness  of  information. 

A  crucial  area  on  which  the  aggregative  comparisons  leave  many  ques- 
tions unanswered  is  whether  better  estimates  of  saving  are  obtained  by 
asking  the  respondent  directly,  or  by  asking  him  for  current  and  past 
holdings,  or  by  conducting  two  interviews  and  asking  for  holdings  at  each 
time.  One  thing  is  clear,  namely,  that  estimating  saving  as  the  difference 
between  income  and  expenditures  tends  to  produce  large  errors.  This  is 
supported  by  most  aggregative  comparisons  in  which  saving  has  been  esti- 

40  For  example,  see  George  Katona  and  J.  B.  Lansing,  "The  Wealth  of  the 
Wealthy,"  Review  of  Economics  and  Statistics,  Vol.  46,  No.  1  (February,  1964), 
pp.  1-13.  In  this  survey  of  families  with  incomes  of  $20,000  or  more,  the  top 
10  percent  of  high-income  people  were  found  to  own  half  of  the  total  assets  of 
the  entire  sample. 
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mated  as  a  residual,  and  it  was  also  demonstrated  very  well  in  an  analytical 
study  by  Modigliani  and  Ando.41 

In  closing  this  chapter,  it  should  be  noted  that  even  with  their  limita- 
tions consumer  savings  surveys  have  made  many  contributions  to  economic 
knowledge.  The  major  such  contributions  made  through  the  availability 
of  such  data  have  been  listed  by  George  Katona  as  follows:42 

( 1 )  The  paramount  importance  of  "contractual  saving"  for  the  majority  of 
individual  families  has  been  demonstrated,  and  it  has  been  established  that  con- 
tractual saving  is  more  stable  and  depends  on  other  factors  than  those  which 
determine  noncontractual  saving. 

(2)  Differences  in  saving  habits  by  age  and  life-cycle  stages  have  been 
indicated. 

(3)  It  has  been  shown  that  during  the  postwar  years  most  of  dissaving  was 
not  associated  with  low  incomes  or  with  adversity  but  was  connected  with  the 
desire  to  acquire  "good  things  of  life";  thus  dissaving  was  a  sign  of  prosperity 
and  contributed  to  it. 

(4)  Owners  and  purchasers  of  United  States  government  savings  bonds, 
life  insurance  policies,  and  new  and  used  cars  have  been  characterized  in  some 
detail,  along  with  borrowers. 

(5)  Information  has  been  collected  on  the  distribution  of  the  burden  of 
both  installment  and  mortgage  debt  among  different  types  of  families  and  on 
the  relation  of  that  burden  to  income. 

(6)  Progress  has  been  made  in  clarifying  the  influence  of  past  income 
changes,  as  well  as  of  liquid  asset  holdings  on  amounts  saved  by  consumers. 

(7)  The  high  degree  of  concentration  of  amounts  saved  among  relatively 
few  families  has  been  indicated. 

Clearly,  therefore,  consumer  survey  data  on  savings  can  be  very  val- 
uable, even  with  their  limitations,  and  would  be  much  more  valuable  if 
these  limitations  could  be  ameliorated.  Especially  in  comparing  changes 
in  distributions  or  other  statistics  over  time,  the  survey  data  can  be  very 
useful  because  the  bias  in  the  data  in  this  respect  appears  to  be  fairly 
constant. 


41  See  Part  I  of  Franco  Modigliani  and  A.  K.  Ando,  "The  'Permanent  Income' 
and  the  'Life  Cycle'  Hypotheses  of  Saving  Behavior:  Comparison  and  Tests," 
Proceedings  of  the  Conference  on  Consumption  and  Saving,  Vol.  2  (Philadelphia: 
University  of  Pennsylvania  Press,  1960),  pp.  51-73. 

42  George  Katona,  "Federal  Research  Board  Committee  Reports  on  Consumer 
Expectations  and  Savings  Statistics,"  Review  of  Economics  and  Statistics,  Vol.  39, 
No.  1  (February,  1957),  pp.  44-45. 


III.   VALIDATION  STUDIES:   DEBT 


This  is  the  first  of  several  chapters  devoted  to  the  reviewing  and  sum- 
marizing of  validation  studies,  mostly  conducted  as  part  of  the  Consumer 
Savings  Project.  These  are  studies  in  which  external  information  serves 
as  a  basis  for  comparing  the  replies  of  the  individual  respondents  on  one 
or  more  aspects  of  their  financial  behavior.  Such  studies  offer  immense 
potential  for  the  development  of  survey  techniques  because  they  are  the 
only  means  of  pinpointing  errors  due  to  response  and  to  nonresponse. 
Moreover,  they  serve  as  a  basis  for  evolving  and  testing  hypotheses  on  the 
causes  of  these  errors.  Unfortunately,  these  studies  are  also  perhaps  the 
most  difficult  to  make.  Not  only  do  many  problems  have  to  be  resolved 
before  the  necessary  external  cooperation  can  be  obtained,  but  other  diffi- 
culties arise  in  the  conduct  of  these  studies  and  in  resolving  problems  of 
comparability  that  arise  in  the  analysis  of  the  data. 

The  present  chapter  is  concerned  with  an  evaluation  of  those  studies 
in  which  one  or  more  forms  of  debt  have  been  validated.  Since  this  is  the 
first  chapter  on  validation,  we  begin  with  a  general  discussion  of  the  man- 
ner in  which  these  studies  have  been  carried  out  and  of  the  types  of  diffi- 
culties encountered.  Then,  a  brief  overview  is  provided  of  the  various 
empirical  validation  studies  that  have  been  conducted,  relating  to  assets  as 
well  as  to  debt.  The  third  part  of  the  chapter  focuses  on  debt  validation 
studies,  with  a  final  section  evaluating  the  results. 

Characteristics  of  a  Validation  Study 

Offhand,  the  conduct  of  a  validation  study  would  seem  to  be  fairly 
straightforward,  once  the  necessary  arrangements  for  securing  the  external 
data  are  made.43  Actually,  however,  numerous  problems  arise  in  the 
course  of  such  a  study,  many  of  which  could  destroy  much  of  the  value  of 
the  operation.   A  brief  review  of  the  principal  such  problems  is  therefore 

43  The  process  of  securing  the  necessary  cooperation  of  external  sources  need 
not  concern  us  here.  Suffice  it  to  note  that  particularly  in  the  case  of  financial 
data  any  arrangements  for  validation  have  to  contain  a  variety  of  safeguards  to 
protect  the  cooperating  institutions  and  to  preserve  anonymity  as  much  as  possible. 
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indicated,  so  that  the  validation  studies  described  in  later  sections  can  be 
more  properly  assessed. 

First  it  is  important,  from  an  analytical  point  of  view,  to  distinguish 
between  two  types  of  validation.  One  type,  which  we  may  call  "ex  ante," 
or  "primary  validation,"  consists  of  using  one  or  more  institutional  sources 
as  a  basis  for  sample  selection.  The  sample  consists  of  holders  of  the  asset 
at  that  source,  usually  selected  on  a  probability  basis,  assuring  representa- 
tion of  all  types  of  holders  and  permitting  unbiased  estimates  of  the  nature 
and  extent  of  omissions  and  reporting  errors  (at  least  for  that  source). 

Second,  we  may  have  "ex  post,"  or  "secondary  validation,"  by  securing 
permission  to  validate  certain  information  after  the  interviews  have  been 
completed.  The  latter  approach  is  the  less  desirable  of  the  two,  because 
only  those  items  can  be  validated  that  are  reported  in  the  interview.  As 
a  result,  it  provides  virtually  no  basis  for  estimation  of  the  extent  of  com- 
plete nonreporting  which,  as  will  be  shown  later,  is  often  the  principal 
source  of  survey  errors.  Nevertheless,  this  approach  does  provide  con- 
siderable information  on  the  nature  and  frequency  of  particular  types  of 
error.  It  can  also  prove  highly  useful  for  supplementary  purposes.  Thus, 
to  date  it  has  not  proved  feasible  to  cross-validate  holdings  on  an  ex  ante 
basis,  because  of  the  widespread  dispersion  of  asset  ownership  and  the 
difficulty  of  making  the  necessary  arrangements.44  More  feasible  is  a 
combination  of  ex  ante  validation  on  one  holding  with  ex  post  validation 
on  others,  which  can  provide  valuable  data  concerning  the  interrelation- 
ship of  reporting  errors. 

An  important  feature  of  a  validation  study  is  that  the  sample  is  selected 
and  the  necessary  data  are  not  usually  obtained  by  members  of  the  survey 
organization,  but  by  members  of  the  cooperating  institution.  Often  the 
names  selected  by  the  institution  are  combined  with  names  from  more 
general  sources,  such  as  city  directories  or  telephone  books.  This  is  a  use- 
ful experimental  feature,  but  at  the  same  time  it  serves  to  introduce  an 
additional  source  of  confusion  when  it  comes  to  matching  information. 

The  fact  that  virtually  all  of  this  preliminary  work  is  carried  out  by 
the  clerical  staff  of  the  cooperating  institution  —  people  who  generally 
have  no  survey  background  —  means  that  biases  due  to  sample  selection 
and  errors  due  to  recording  mistakes  are  more  likely  to  occur  than  would 
otherwise  be  the  case.  This  is  particularly  true  since  this  work  is  usually 
assigned  to  the  institution's  staff  in  addition  to  their  regular  duties,  and 
they  are  generally  under  pressure  to  complete  the  work  quickly.   The  fact 

44  For  a  discussion  of  the  implications  of  matching  errors,  see  John  Neter, 
E.  S.  Maynes,  and  R.  Ramanathan,  "The  Effect  of  Mismatching  on  the  Measure- 
ment of  Response  Errors,"  Proceedings  of  the  Social  Statistics  Section  of  the 
American  Statistical  Association  (Washington:  American  Statistical  Association, 
1964),  pp.  1-8. 
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that  this  information  is  supplied  on  a  one-time  basis  and  that  rechecks 
are  not  always  possible  does  not  make  the  situation  any  easier.45 

The  field  work  on  such  studies  also  tends  to  be  more  difficult  than  is 
usually  the  case.  The  principal  reason  is  that  the  interview  generally  has 
to  be  conducted  not  only  with  a  specific  family  at  a  specific  address,  but 
with  a  specific  member  of  that  family.  In  some  cases,  insufficient  infor- 
mation exists  for  the  member  to  be  specified,  a  problem  that  becomes  all 
the  more  difficult  because  the  sources  of  the  names  and  the  real  purposes 
of  the  study  cannot  be  divulged  to  the  interviewers.  The  result  is  con- 
siderable leeway  for  mistaken  identification. 

No  less  difficult  than  the  preceding  stages  is  comparison  of  the  infor- 
mation supplied  by  the  respondent  with  that  available  from  the  external 
source.  At  this  stage,  all  the  possibilities  for  faulty  matching  or  identifi- 
cation come  together.  Essentially,  there  would  seem  to  be  three  principal 
sources  of  error.  One  source  is  in  identifying  the  respondent.  The  inter- 
view might  have  been  conducted  with  a  different  family  of  the  same  name 
(something  not  at  all  unlikely  when  interviewing  in  a  large  apartment 
building  and  dealing  with  fairly  common  names)  or  the  respondent  who 
should  have  been  interviewed  was  not  available  for  one  reason  or  another. 
For  example,  a  son  who  moves  out  of  his  parents'  home  may  still  have 
many  of  his  assets  and  debts  recorded  at  the  previous  address. 

A  second  source  of  error  lies  in  faulty  identification  of  the  holding 
being  validated.  This  is  particularly  true  of  assets  and  debts  of  which 
more  than  one  are  held  by  many  consumer  units.  For  example,  many 
families  have  more  than  one  savings  account,  more  than  one  debt,  more 
than  one  life  insurance  policy,  or  more  than  one  bond  or  share  of  stock. 
Since  the  external  source  consists  invariably  of  a  particular  institution  or 
group  of  institutions,  the  matching  has  to  be  carried  out  by  individual 
holdings.  Since  these  holdings  often  are  not  fully  described,  the  matching 
problem  may  become  partly  a  matter  of  guesswork.  Thus,  only  one  of  two 
debts  on  a  car  may  be  known  from  an  external  source.  If  the  respondent 
reports  two  debts,  and  if  the  name  of  the  institution  is  not  obtained,  there 
may  be  little  basis  for  deciding  which  debt  corresponds  to  the  external 
report.  Even  if  the  name  of  the  institution  is  reported  by  the  respondent, 
there  is  so  much  similarity  of  names  in  the  financial  world  that  reporting 
of  an  inaccurate  name  is  always  possible. 

Third,  there  is  the  matching  of  holdings  in  time.  If  the  external  report 
is  obtained  just  prior  to  the  payment  of  an  additional  installment  on  a 
debt  and  the  respondent  reports  the  amount  of  the  debt  after  this  pay- 
ment, differences  will  result  solely  for  this  reason.   The  problem  is  partic- 

4r'  Institutions  will  rarely  allow  outsiders  to  work  with  their  records,  not  to 
mention  the  fact  that  after  a  relatively  short  time  many  records  are  put  in  dead 
storage  and  are  virtually  inaccessible. 
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ularly  perplexing  in  the  case  of  holdings  where  there  is  a  large  "float,"  as 
in  the  case  of  checking  accounts.  The  balance  reported  by  the  respondent 
and  the  balance  obtained  from  the  external  source  may  differ  substantially, 
yet  both  may  be  fully  accurate. 

Many  of  these  sources  of  error  can  be  remedied  if  the  validation  study 
is  carried  out  as  part  of  a  panel  operation.  Even  then,  however,  problems 
remain  and,  as  will  be  noted  later,  discrepancies  can  arise  for  many  dif- 
ferent reasons.  The  central  point  that  emerges  from  this  discussion,  there- 
fore, is  that  discrepancies  do  not  automatically  indicate  response  error. 
Indeed,  as  we  shall  soon  see,  many  discrepancies  uncovered  in  these  studies 
are  not  attributable  to  inaccurate  reporting  at  all,  but  rather  serve  to  high- 
light the  unique  characteristics  of  the  particular  source  from  which  the 
data  are  obtained. 

Background  Information  on  Empirical  Studies 

General  information  on  the  nature  and  objectives  of  past  financial 
validation  studies  is  presented  in  Table  7.  For  each  study  the  table  shows 
the  sponsoring  organization,  the  date  of  the  field  interviews,  the  scope 
and  nature  of  the  sample  and  of  the  population  studied,  and  the  items 
validated.  In  the  latter  case,  holdings  validated  on  an  ex  ante  basis  are 
distinguished  from  holdings  validated  on  an  ex  post  basis.  In  most  cases, 
only  one  item  of  information  was  validated.  As  is  evident  from  the  table, 
to  date  four  organizations  have  been  responsible  for  the  financial  valida- 
tion studies.  Eight  studies  were  undertaken  as  part  of  the  Consumer  Sav- 
ings Project,  of  which  four  were  subcontracted  to  the  Survey  Research 
Center;  the  other  four  were  conducted  from  the  project  headquarters  at 
the  University  of  Illinois. 

In  addition  to  providing  general  background  information  on  each  of 
these  studies,  the  table  also  brings  out  various  salient  features.  One  inter- 
esting characteristic  is  that  all  of  these  studies  are  relatively  recent,  none 
having  been  undertaken  before  1954  and  most  of  them  having  been  con- 
ducted during  the  later  1950's  and  early  1960's.  Second,  most  of  these 
studies  have  been  of  limited  scope.  Few  of  the  samples  exceeded  300. 
This  was  partly  due  to  limitation  of  financial  resources  and  partly  because 
of  the  feeling  that  the  most  information  per  dollar  in  this  relatively  un- 
explored area  could  be  obtained  by  a  sequential  series  of  small,  pilot 
studies  rather  than  by  two  or  three  large  studies.  Also,  a  number  of  small 
studies  enabled  a  variety  of  survey  techniques  to  be  evaluated,  some  by 
experimental  design  and  others  by  observation.  The  sequential  nature  of 
the  operation  made  possible  the  use  of  results  from  previous  studies  in 
selecting  promising  new  approaches  for  testing  and  in  the  construction 
of  appropriate  testing  procedures. 
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A  third  characteristic  is  that  increasing  focus  has  been  placed  on  secur- 
ing heavy  representation  from  the  high-income  levels  or  from  large  holders 
of  the  asset  or  debt  being  validated.  This  is  not  surprising  in  view  of  the 
findings  of  the  aggregation  studies  covered  in  the  previous  chapter  that 
these  groups  might  account  for  most  of  the  errors.  The  validation  studies 
have  not  contradicted  this  finding,  although  the  situation  does  not  seem  to 
be  as  clear-cut  as  was  originally  suspected.  The  fact  remains  that  even  if 
these  groups  do  not  account  for  a  more-than-proportionate  share  of  the 
errors,  the  size  of  their  financial  holdings  warrants  special  emphasis  on 
them. 

Fourth,  most  of  the  studies  have  been  conducted  in  urban  areas.  This 
has  been  partly  because  of  cost  considerations,  partly  because  certain  assets 
are  concentrated  heavily  in  urban  areas,  and  partly  because  survey  errors 
appear  to  be  more  frequent  in  such  areas. 

Fifth,  most  of  the  studies  have  been  one-time  surveys.  In  other  words, 
the  objective  has  been  to  obtain  relevant  financial  information  in  a  single 
interview,  with  no  attempt  to  obtain  information  on  the  same  assets  and 
debts  at  some  later  time.  The  latter  approach,  which  when  extended  fully 
becomes  a  continuous  panel  operation,  has  been  undertaken  only  in  some 
of  the  studies  of  the  Consumer  Savings  Project.  In  one  of  the  other  studies, 
however,  sample  members  were  interviewed  roughly  nine  months  apart 
and  asked  for  current  holdings  each  time. 

Sixth,  most  of  the  validation  studies  focused  either  on  debt  or  on 
savings  accounts.  The  reason  for  the  former  was  primarily  a  matter  of 
expediency.  The  necessary  arrangements  were  relatively  easy,  and  there 
is  generally  not  much  difficulty  in  inducing  people  to  talk  about  their 
debts.  Hence,  this  asset  fits  in  well  at  the  exploratory  stage.  The  three 
pilot  tests  undertaken  as  part  of  Study  S2  are  a  good  example  of  this  type 
of  approach. 

On  the  other  hand,  validation  of  savings  accounts  appears  desirable  for 
much  the  opposite  reason.  As  will  be  shown  later,  it  is  one  of  the  more, 
if  not  the  most,  sensitive  assets.  People  generally  are  more  reluctant  to 
discuss  their  savings  accounts  than  any  other  aspect  of  financial  (or  other) 
behavior.  As  a  result,  it  provides  an  acid  test  of  the  reliability  of  a  partic- 
ular set  of  survey  techniques.  In  addition,  unlike  similarly  sensitive  assets 
such  as  common  stock,  it  is  widely  held,  so  that  estimates  can  be  obtained 
of  the  reliability  of  survey  techniques  on  all  different  types  of  people. 

Seventh,  cross-validation  —  the  validation  of  more  than  one  type  of 
holding  per  sample  member  —  is  comparatively  rare.  Although  feasible 
theoretically,  in  practice  the  necessary  collation  of  institutional  records  is 
often  impossible  to  arrange.  Different  types  of  institutions  tend  to  keep 
their  records  in  different  ways.    This,  combined  with  restricted  access  to 
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records  and  restricted  time  provided  by  institutional  staff,  makes  the 
matching  of  records  for  validation  purposes  very  difficult.  In  the  only 
instance  where  more  than  one  asset  served  as  an  original  validation  source, 
Study  Fl,  all  the  data  came  from  the  same  set  of  records. 

An  alternative  procedure,  followed  in  Study  P3,  was  to  use  different 
types  of  records  as  original  validation  sources,  each  accounting  for  a  sep- 
arate portion  of  the  sample.  After  the  interviews  were  carried  out,  group- 
validation  was  attempted  both  among  these  assets  and  debts  and  among 
certain  others  indicated  in  the  table. 

Although  not  recorded  in  the  table,  it  is  pertinent  to  note  that  all  of 
these  studies,  with  the  exception  of  S2c  and  N,  covered  in  the  interview 
situation  not  only  the  items  being  validated  but  also  a  number  of  other 
assets  and  debts.  In  the  cases  of  the  Federal  Reserve  and  the  Consumer 
Savings  Project  studies,  the  entire  range  of  assets  and  debts  was  covered 
in  most  of  the  interviews.46 

More  detailed  information  on  the  interviews  and  on  the  content  of  the 
individual  studies  will  be  presented  in  later  sections.  Certain  details,  how- 
ever, can  not  be  revealed  in  view  of  the  confidential  arrangements  under 
which  these  studies  were  conducted.  For  this  reason,  no  information  is 
given,  other  than  that  shown  in  Table  7,  on  the  location  of  the  studies, 
the  sources  of  external  information,  or  the  manner  in  which  the  validation 
data  were  obtained. 

Debt 

The  remainder,  and  largest  part,  of  this  chapter  is  devoted  to  a  review 
of  the  principal  findings  of  known  studies  of  debt  validation.  Primarily 
the  highlights  are  presented  for  those  studies  that  have  already  been  pub- 
lished, with  references  to  the  original  sources  for  the  supporting  evidence. 
More  detail  is  given  on  studies  reported  here  for  the  first  time,  namely, 
three  studies  of  the  Consumer  Savings  Project. 

National  Survey  of  New  Car  Debt 

The  first  known  study  validating  debt  was  carried  out  in  1956  by  the 
Federal  Reserve  Board.  As  part  of  a  comprehensive  investigation  of  con- 
sumer installment  credit  in  1956,  the  Federal  Reserve  Board  undertook  a 
national  survey  of  means  of  financing  new  car  purchases.47  In  connection 
with  this  survey,  information  on  new  car  purchases  and  on  the  conditions 
under  which  loans  were  granted  was  sought  from  new  car  purchasers  as 
well  as  from  lenders.  As  a  result,  it  proved  feasible  to  match  certain  data 
from  both  sources. 


46  See,  for  example,  the  questionnaire  in  Appendix  A. 

"  The  results  of  this  study  were  published  in  Consumer  Installment  Credit, 
Part  IV,  Financing  New  Car  Purchases  (Washington:  Board  of  Governors  of  the 
Federal  Reserve  System,  1957). 
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The  original  sample  consisted  of  a  random  selection  of  550  new  car 
registrants  in  each  of  the  24  months  in  1954-55  from  112  stratified  counties 
in  41  states.  About  1,000  registrations  were  in  the  names  of  businesses  and 
governments  and  were  therefore  eliminated.  Of  the  remaining  registra- 
tions, lender  interviews  were  sought  for  all  new  car  purchases  which  in- 
volved credit,  and  buyer  interviews  were  sought  for  a  randomly  selected 
half  of  the  roughly  12,000  registrations. 

The  validation  was  restricted  to  those  registrations  for  which  a  buyer 
interview  was  sought,  for  which  conclusive  information  was  obtained 
from  the  lender  that  the  new  car  purchase  involved  credit,  and  for  which 
there  seemed  little  doubt  of  the  accuracy  of  the  data  obtained  from  the 
lender.  Altogether,  this  amounted  to  about  6,500  registrations. 

The  actual  interviews  were  conducted  during  June  and  July  of  1956. 
The  questionnaire  focused  heavily  on  new  car  purchases,  on  the  reasons 
and  circumstances  under  which  the  purchases  were  made,  and  on  the  use 
of  credit  and  the  conditions  under  which  credit  was  obtained.  On  the 
average  the  interviews  lasted  45  minutes. 

Considerable  pressure  was  placed  on  the  buyers  to  cooperate,  and  they 
were  sent  an  advance  letter  mentioning  that  "the  study  was  being  made 
at  the  request  of  the  President  and  interested  committees  of  Congress." 
Nevertheless,  the  overall  response  rate  on  the  buyer  interviews  was  only 
72  percent  of  the  6,444  attempted.  Most  of  the  nonresponse  rate  (15  per- 
cent of  the  28  percent)  was  due  to  the  long  time  lag  between  the  date 
of  the  new  car  registration  and  the  date  of  the  interview,  by  which  time 
many  of  the  buyers  had  moved  out  of  the  area  and  could  not  be  located. 
Noncontacts  of  people  living  in  the  area  were  responsible  for  roughly  half 
of  the  remaining  nonresponse,  with  the  other  half  resulting  from  refusals.48 

Matching  lender  reports  were  obtained  from  90  percent  of  the  6,389 
credit  purchases  for  which  such  information  was  sought.  Roughly  one- 
third  of  the  nonresponse  was  due  to  refusals  and  the  other  two-thirds  to 
the  inability  to  locate  the  necessary  records,  which  in  many  cases  had  been 
discarded  because  the  accounts  were  already  closed.  As  a  result,  roughly 
3,000  lender  interviews  were  available  which  involved  the  use  of  credit 
for  which  a  buyer  interview  was  also  attempted. 

These  interviews  are  the  basis  for  the  validation  analysis,  a  summary 
of  which  is  shown  in  Table  8.  The  four  items  of  information  for  which 
data  are  presented  in  this  table  served  as  the  basis  for  the  analysis  of  re- 
sponse and  nonresponse  errors.  These  are  the  new  car  price,  the  total 
amount  borrowed,  the  amount  of  the  principal  loan  (which  would  be  the 
same  as  the  total  loan  if  there  were  only  one  loan),  and  the  monthly  pay- 
ment.  The  approach  taken  in  this  table  is  similar  to  that  followed  in  the 

"Ibid.,  p.  122. 
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Table  8.  Response  and  Nonresponse  Errors  in  National  Survey 
of  New  Car  Buyers  (Study  Fl ) ,  1956 


Item 

New  car 
price 

Total 
loan 

Principal 
loan 

Monthly 
payments 

1 .  Sample  size 

1,655 

$2,886 

101 
106 
108 

99 

1 
1.3 

10.1 
76.4 
13.5 
28.2 
26.8 
45.0 

1,918 
$2,038 

107 
102 
105 

98 

7 

4.0 

6.2 

74.7 

19.1 

28.1 

78.0 

-6.1 

1,836 
$1,735 

106 

102 
105 

98 

b 

4.9 
11.5 
64.9 
23.6 
28.5 
48.7 
22.8 

1,814 

$76 

2.  Average  "actual"  value  of 

validated  item8 

Nonrespondents :  sample  aver- 
ages as  percentage  of 
"actual"  value 
3.       Noneligibles 

108 

4.       Noncontacts 

105 

5.       Refusals 

107 

6.  Respondents:   sample  average 
as  percentage  of  "actual" 

97 

7.  Nonreporters:    percentage  of 

respondents  not  mention- 
ing item 

Reporters:   response  errors 

8.  Average  percent  of  error0 ..  . 

9.  Percentage  overstating  by 

10.  Percentage  within  10 

1 1 .  Percentage  understating  by 

10  percent  or  more 

Allocation  of  error  by  percent 

12.  To  nonrespondents 

13.  To  nonreporting  of 

respondents 

b 

0 

6.0 

88.4 

5.6 

29.3 

141.7 

14.       To  inaccurate  reporting  of 

respondents 

-71.0 

a  The  "actual"  values  in  all  instances  are  estimated  from  the  lender  reports. 
b  Not  available,  but  reported  to  be  between  1  and  7  percent. 
c  Lender  report-respondent  report 

Lender  report 
Source:    Arthur  L.  Broida,  "Consumer  Surveys  as  a  Source  of  Information  for  Social  Account- 
ing —  The   Problems,"    paper   delivered   at   the   Conference   on   Research   on   Income   and   Wealth, 
1959,    and   published   in   Studies   in   Income   and   Wealth,   Vol.    26,    NEBR    (Princeton:     Princeton 
University  Press,  1962),  Part  IV. 


original  study:  the  lender  figure  was  considered  to  be  the  correct  figure 
for  analysis  unless  there  was  clear  evidence  to  the  contrary.  In  effect, 
therefore,  these  figures  are  accepted  as  the  "actual"  figures,  and  deviations 
of  the  respondent  reports  were  computed  using  these  figures  as  the  base. 
Such  deviations,  converted  for  each  segment  of  the  sample  as  a  ratio, 
are  shown  in  lines  3-6  of  Table  8.  As  is  evident  from  these  data,  neither 
the  errors  due  to  nonresponse  (lines  3,  4,  and  5)  nor  the  response  errors 
(line  6)  are  particularly  large  —  at  least,  they  are  not  comparable  with 
the  error  magnitudes  encountered  in  the  previous  chapter.  Nevertheless, 
certain  patterns  are  apparent.  Thus,  there  is  a  clear  tendency  for  the  fig- 
ures for  the  noncontacts  and  the  refusals  to  exceed  the  overall  sample 
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average.  These  differences,  which  are  statistically  significant  at  the  .05 
probability  level,  are  most  pronounced  for  the  refusals  and  for  the  non- 
eligibles.  (In  the  latter  case,  however,  it  is  puzzling  to  note  how  much 
smaller  the  deviation  of  the  new  car  price  is  than  the  deviations  of  the 
other  three  items.) 

On  the  other  hand,  the  reports  of  the  respondents  tend  to  understate 
slightly,  although  consistently,  the  overall  sample  averages  for  all  four 
items.  Part  of  the  reason  for  this  understatement  lies  in  the  failure  of  some 
respondents  to  report  the  purchase  or  the  debt  at  all.  The  extent  of  such 
nonreporting,  shown  in  line  7  of  Table  8,  varies  from  7  percent  of  the 
respondents  in  the  case  of  the  total  loan  to  barely  1  percent  for  the  pur- 
chase of  the  new  car  itself. 

Data  on  the  other  component  of  response  error  —  errors  in  those  debts 
that  were  reported  —  are  provided  in  line  8  of  the  table.  The  error 
figures  in  this  line  are  restricted  to  those  instances  in  which  information 
on  the  particular  item  was  obtained  on  a  comparable  basis  from  both  the 
buyer  and  the  lender.  As  the  data  in  this  line  show,  the  average  monthly 
payment  coincided  exactly  (at  least  to  the  nearest  dollar)  with  the  figure 
reported  by  the  lender.  Furthermore,  no  appreciable  bias  in  the  distribu- 
tion of  errors  is  perceptible,  since  roughly  6  percent  of  the  respondents 
understated  the  actual  monthly  payment  by  10  percent  or  more  and  the 
same  proportion  overstated  the  figure  by  this  margin. 

On  the  other  three  items,  however,  the  response  errors  are  in  the  di- 
rection of  understatement.  The  response  error  on  the  purchase  price  of 
the  new  car  is  the  lowest  of  the  three,  coming  within  1.3  percent  of  the 
figure  reported  by  the  lender.  The  tendency  toward  understatement  is 
brought  out  by  the  positive  sign  of  this  error  as  well  as  by  the  fact  that 
the  proportion  of  respondents  erring  by  10  percent  or  more  on  the  pur- 
chase price  is  considerably  higher  than  in  the  case  of  the  monthly  payment. 
Nevertheless,  more  than  three-fourths  of  the  respondents  were  within  10 
percent  of  the  lender's  figure. 

Appreciably  higher  errors  were  obtained  on  the  amounts  of  the  loans, 
and  in  both  instances  pronounced  tendencies  toward  understatement  are 
apparent.  Thus,  the  amount  of  the  total  loan  was  understated  on  the 
average  by  4  percent,  and  the  principal  loan  was  understated  by  almost 
5  percent.  In  addition,  almost  one-fifth  of  the  respondents  understated  the 
total  loan  by  10  percent  or  more  and  the  same  was  true  of  almost  one- 
fourth  of  the  respondents  for  the  principal  loan.  In  contrast,  overstate- 
ments were  less  than  half  as  frequent. 

As  Broida  points  out,49  the  much  higher  accuracy  of  a  report  of  the 

49  Broida,  op.  cit.,  p.  66. 
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monthly  payment  is  not  surprising,  considering  the  fact  that  this  figure 
is  brought  to  the  attention  of  the  debtor  every  month.  On  the  other  hand, 
the  price  of  the  new  car  is  more  or  less  a  matter  of  past  history,  the  pur- 
chase having  taken  place  anytime  between  6  months  and  30  months  prior 
to  the  interview.  The  same  is  true  of  the  amounts  of  the  loans.  The  still 
larger  errors  in  those  cases  could  be  due  to  various  reasons.  One  is  that 
some  people  might  have  hesitated  to  report  the  full  extent  of  their  borrow- 
ing. Others  might  have  by  this  time  more  or  less  unconsciously  ration- 
alized the  amount  of  debt  to  a  lower  level.  In  still  other  cases,  lack  of 
comparability  between  the  reports  of  the  lender  and  those  of  the  buyer 
might  have  been  responsible  (although  it  is  not  clear  why  this  should  lead 
to  a  persistent  tendency  toward  understatement) . 

The  last  three  lines  of  the  table  (lines  12-14)  provide  estimates  of  the 
relative  importance  of  these  different  types  of  errors.  These  estimates  are 
necessarily  rough,  partly  because  all  the  required  data  were  not  available 
and  partly  because  of  the  various  ways  in  which  such  estimates  can  be 
made.  Nevertheless,  these  estimates  furnish  striking  indications  of  the 
relative  importance  of  the  different  types  of  errors,  which  are  not  likely 
to  be  altered  significantly  by  changes  in  the  basic  assumptions.  Further- 
more, this  general  approach  toward  allocating  errors  enables  useful  com- 
parisons to  be  made  between  different  studies. 

For  the  present  purposes,  we  are  concerned  with  segregating  the  error 
in  the  sample  estimate  of  the  mean  value  of  each  of  the  four  validated 
financial  items  among  three  causes:  error  due  to  inability  to  interview  all 
sample  members,  error  due  to  failure  of  those  interviewed  to  report  the 
validated  item,  and  error  due  to  inaccuracies  in  the  amounts  given  by 
those  respondents  who  did  report  the  validated  item.  Let  us  designate  by 
N  the  total  sample;  N8,  the  number  of  nonrespondents;  N0,  the  respond- 
ents not  reporting  the  item;  and  N1}  the  respondents  who  did  report  the 
item.  A-,,  denotes  the  corresponding  "true"  average  values  of  the  vali- 
dated item  for  segment  i,  and  Xi  is  the  computed  average  for  that  seg- 
ment based  on  the  interview  data.  Then,  the  actual  average  value  of  that 
item  for  the  entire  sample  is: 

,aiv                                A       NSAS  +  NqAq  +  NiAj 
{?•*)  A  =  jy , 

and  the  corresponding  interview-derived  estimate  of  A  is: 

(3.2)  Estimate  of  A  =  X  = -^f • 

N 

In  practice,  no  figures  are  reported  for  X0,  which  is  identically  zero. 
Also,  the  value  of  the  item  under  study  is  not  known  for  the  nonrespon- 
dents and  is  therefore  estimated,  implicitly  or  otherwise,  from  the  informa- 
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tion  supplied  by  the  respondents,  usually  as  the  mean  value  of  the  latter.50 
Hence,  we  take  as  the  estimate  of  As  the  value  of  Xr,  the  mean  value 
reported  by  the  respondents  (=N1X1/  (N0  +  Nx)).  From  a  computa- 
tional point  of  view,  therefore,  the  estimate  of  A  becomes: 

/QQ,  p    ,.  .  ,   A  v  NsXr  +  NlX! 

(3.3)  Estimate  of  A  =  A  = ^ . 

The  error  (E)  in  this  estimate  is,  from  the  above: 

(3.4)  E  =  A  -  X  =  [Ns(As  -  Xr)  +  N0A0  +  Ni(A,  -  X{)]/N. 

The  right-hand  side  of  this  equation  partitions  the  total  error  into  the 
three  desired  components,  and  the  percentage  distributions  of  these  com- 
ponents are  shown  in  lines  12-14  of  Table  8.51  The  results  indicate  that 
with  the  exception  of  new  car  price,  the  principal  source  of  error  in  this 
study  was  nonreporting  by  respondents,  even  though  such  nonreporting 
was  low.  In  the  case  of  new  car  price,  nonreporting  was  virtually  absent, 
which  explains  the  relatively  low  error  attributed  to  that  item  for  this 
cause. 

Errors  due  to  faulty  estimates  of  the  mean  value  of  the  validated  item 
for  the  nonrespondents  accounted  for  roughly  30  percent  of  the  total  error, 
in  all  cases  tending  to  reduce  the  sample  estimate,  as  was  noted  previously. 
The  effect  of  inaccurate  reports  of  respondents  varied  substantially  from 
one  item  to  another.  For  the  total  loan  and  for  monthly  payments,  inac- 
curate reports  partially  offset  errors  for  the  other  two  causes,  particularly 
in  the  case  of  the  monthly  payments.  On  the  other  hand,  for  new  car 
price  and  for  principal  loan,  inaccurate  reports  served  to  add  to  the  total 
error. 

All  things  considered,  the  results  of  this  study  did  seem  to  indicate 
that,  except  for  cases  in  which  nonreporting  was  virtually  nil,  it  repre- 
sented the  largest  part  of  the  field  errors.  The  fact  remains,  however,  that 
this  survey  was  carried  out  under  special  circumstances  and  was  restricted 
to  an  easily  discussed  aspect  of  financial  behavior. 

Let  us  now  consider  the  extent  to  which  the  biases  shown  in  Table  8 
distort  the  reliability  of  the  estimate  of  the  mean  rather  than  the  estimate 
itself.  To  do  so,  we  make  use  of  Equation  (1.3)  from  Chapter  I  to  com- 
pute the  true  variance  of  the  mean  as  the  sum  of  the  variance  of  the  mean 

50  A  somewhat  more  refined  approach  is  to  estimate  the  value  for  each  non- 
respondent  as  the  mean  value  of  respondents  having  the  same  set  of  prior- 
determined  relevant  characteristics.  Such  a  modification  could  be  included  in  the 
present  framework,  but  is  unnecessary  for  the  present  purposes. 

51  This  technique  may  be  used  to  partition  the  total  error  into  any  number  of 
relevant  components,  provided  that  the  necessary  data  are  available,  including 
allowance  for  incorrect  estimates  of  the  population  distribution  (in  this  case,  the 
Ni's).   It  will  be  used  for  such  purposes  in  later  chapters. 
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of  the  observed  values  and  the  square  of  the  bias.  The  bias  is  estimated  as 
the  difference  between  the  average  value  obtained  from  the  lender  reports 
(line  2  of  Table  8)  and  the  average  figure  computed  from  the  data  sup- 
plied by  the  respondents. 

The  results  of  these  computations  are  shown  in  the  following  tabula- 
tion together  with  estimates  of  the  "bias  multiplier,"  k{E/oy)  : 

Standard  error  of  mean 
Bias  k  Apparent      True  value 

New  car  price $78  4.5  $17.28       $79.89 

Total  loan 128  8.1  15.83         128.97 

Principal  loan 128  9.6  13.38         128.70 

Monthly  payments 2  3.0  .66  2.10 

Clearly,  the  bias  term,  although  small,  is  still  many  times  larger  than 
the  estimated  variance  of  the  mean.  Indeed,  the  large  size  of  the  sample 
in  this  case  acts  to  magnify  the  importance  of  the  bias,  serving  to  reduce 
sampling  errors  to  almost  minute  proportions  while  having  virtually  no 
effect  on  the  bias  itself.  As  a  result,  the  apparent  standard  error  of  the 
mean  for  the  buyer  reports,  reproduced  from  the  original  source,52  actually 
understates  tremendously  the  true  standard  error  of  the  mean.  Indeed, 
in  this  suvey  the  principal  effect  of  the  bias  is  to  distort  completely  esti- 
mates of  reliability  while  producing  little  bias  in  the  estimates  themselves. 

We  may  next  move  one  step  further  and  ask :  what  proportion  of  the 
apparent  standard  error  of  the  mean  is  attributable  to  response  error? 
In  other  words,  what  would  be  the  true  standard  error  of  the  mean  in  the 
absence  of  response  error?  Although  data  are  lacking  to  make  exact  cal- 
culations, on  the  basis  of  Equation  (1.7)  on  page  14,  there  is  little  doubt 
that  in  this  case  the  difference  is  virtually  nil.  The  biases  noted  in  Table 
8  are  so  small  that  the  increase  in  sampling  variance  due  to  response  error 
is  also  likely  to  be  very  small.53    Hence,  the  principal  effect  of  response 

52  Broida,  op.  cit.,  Table  3,  p.  372. 

53  This  is  based  on  the  following  reasoning.  As  noted  previously,  the  true  values 
and  the  response  errors  tend  to  be  positively  correlated.  Using  the  same  terminology 
as  in  Chapter  I   (pp.  11  and  12),  assume  as  a  rough  approximation  that: 

ei  =  gxi,  where  0  <  g  <  1 ,  and 

<r<?  =  g2<rx2. 

Substituting  in  Equation  (1.5)  on  page  13: 

«V=»*(1  +g2  +  2grxe). 

To  judge  from  Table  8,  a  liberal  estimate  of  g  (for  present  purposes)  is  .05.  Similarly, 
r  is  hardly  likely  to  exceed  .5,  based  on  similar  correlations  presented  in  this  and  later 
chapters.    Substituting: 

0-2  =  0-2  (t   +.0525). 

Thus  the  increase  in  the  estimated  standard  error  due  to  response  error  is  not  likely 
to  exceed 

V-0525  ffx,  or  2.3  percent. 
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error  on  the  reliability  of  the  estimates  is  to  distort  completely  the  stand- 
ard error  estimate,  and  any  confidence  intervals  based  on  it,  rather  than 
to  inflate  the  estimate. 

Further  light  on  the  nature  of  the  nonsampling  errors  has  been  ob- 
tained by  chi-square  tests  carried  out  in  turn  to  ascertain  the  extent  to 
which  different  variables  were  associated  with  error  in  the  amount  of  total 
loan  and  in  the  amount  of  principal  loan;  these  variables  reflected  re- 
spondent characteristics,  interviewer  characteristics,  and  characteristics  of 
the  interview,  and  the  dependent  variable  in  all  of  these  tests  was  the  per- 
centage error  in  the  amount  of  the  loan.54  Percentages  rather  than  abso- 
lute numbers  were  used  because  of  a  pronounced  tendency  for  the 
absolute  error  to  increase  with  the  amount  of  the  loan. 

In  all  of  these  tests  the  percentage  errors  were  subdivided  in  three 
categories :  underreported  by  more  than  1  percent,  accurate  within  1  per- 
cent, and  overreported  by  more  than  1  percent.  In  the  light  of  later 
studies,  this  distinction  at  1  percent  seems  arbitrarily  fine,  and  it  may  have 
served  to  obscure  characteristics  of  those  making  really  substantial  errors. 
The  fact  remains,  however,  that  a  high  degree  of  accuracy  was  obtained 
in  this  study,  which  renders  this  1  percent  boundary  line  a  much  more 
sensitive  discriminant  of  error  than  it  otherwise  would  have  been. 

The  tests  revealed  a  number  of  characteristics  to  be  related  significantly 
(at  the  .05  probability  level)  to  one  or  the  other  of  these  variables.  For 
total  loan,  the  following  variables  were  significant: 

Race.  Negroes  were  accurate  more  frequently  than  whites.  It  is  not 
clear,  however,  whether  or  not  this  might  be  due  to  income  and  occupa- 
tion differentials  rather  than  to  race  as  such. 

Consultation  of  records.  Many  more  of  those  consulting  records  were 
accurate  within  1  percent  than  of  those  who  did  not  consult  records. 

Occupation  of  head.  Those  in  laboring  occupations  were  the  most  ac- 
curate and  the  self-employed  were  the  least  accurate. 

Type  of  lender.  Borrowers  from  sales  finance  companies  were  more 
frequently  accurate  than  borrowers  from  banks  or  trust  companies.  Here, 
too,  the  difference  may  reflect  more  basic  factors,  such  as  the  differing 
extent  of  patronage  of  these  institutions  by  various  economic  groups. 

Education  of  interviewer.  Oddly  enough,  interviewers  who  had  at- 
tended but  not  completed  college  received  a  higher  proportion  of  accurate 
responses  than  interviewers  with  either  more  or  less  education. 


"Rikumo  Ito,  "A  Multivariate  Analysis  of  Response  Errors,"  unpublished 
Ph.D.  dissertation,  University  of  Illinois,  1962;  and  "An  Analysis  of  Response 
Errors:  A  Case  Study,"  Journal  of  Business,  Vol.  36,  No.  4  (October,  1963), 
pp.  440-47. 
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Age  of  interviewer.  Those  in  the  36-45  age  group  produced  the  highest 
frequency  of  accurate  responses. 

Only  three  variables  proved  significantly  associated  with  error  in  the 
report  of  the  principal  loan : 

Consultation  of  records.  The  same  marked  relationship  was  apparent 
as  with  total  loan. 

Region.  Those  in  the  Northeast  and  in  the  North  Central  areas  tended 
to  be  accurate  more  frequently. 

Date  of  interview.  A  lower  frequency  of  accurate  reports  was  pro- 
duced on  later  interviews.  This  is  not  surprising  because  later  interviews 
are  generally  either  with  people  who  could  not  be  contacted  previously 
or  with  those  who  have  for  one  reason  or  another  put  off  being  interviewed 
and  are  likely  to  be  less  cooperative. 

Particularly  noteworthy  is  the  fact  that  virtually  all  of  the  statistically 
significant  relationships  were  characterized  by  a  single  attribute:  the  dif- 
ferences occurred  entirely  among  the  percentages  of  respondents  that  were 
accurate  and  that  underreported.  Virtually  no  differences  were  observed 
in  the  percentage  of  respondents  overreporting  either  principal  loan  or 
total  loan.  This  would  suggest  that  in  this  particular  study  both  the  in- 
tentional as  well  as  the  unconscious  tendencies  to  misreport  were  entirely 
in  the  direction  of  understating  the  true  figure.  Since,  as  was  noted 
previously,  the  frequency  of  underreporting  was  substantially  higher  than 
the  frequency  of  overreporting,  the  result  was  an  understatement  of  the 
aggregate,  albeit  by  a  small  percentage. 

Variables  not  significantly  related  to  the  frequency  of  error  in  either 
case  included  liquid  assets,  income  of  the  respondent,  interviewer  attend- 
ance at  a  training  session,  whether  the  interviewer  was  a  supervisor  or  not, 
and  the  number  of  months  prior  to  the  interview  that  the  car  had  been 
purchased. 

Only  one  variable,  consultation  of  records,  was  associated  significantly 
with  the  frequency  of  error  in  both  loan  amounts.  However,  both  occupa- 
tion of  head  and  age  of  interviewer  were  almost  significant  at  the  .05  level 
with  regard  to  principal  loan,  and  region  was  almost  significant  with  re- 
gard to  total  loan. 

Exploratory  Studies  of  New  Car  Debt 

With  funds  supplied  by  the  Consumer  Savings  Project,  the  Survey 
Research  Center  of  the  University  of  Michigan  carried  out  during  1957 
three  exploratory  studies  on  the  nature  of  response  errors  on  new  car 
debt.  All  of  these  studies  were  of  limited  scope,  each  designed  to  test 
particular  hypotheses.  The  first  one,  Study  S2a,  consisted  of  25  interviews 
in  three  Michigan  counties  during  January-February,  1957,  with  people 


Validation  Studies:    Debt  57 

who  had  incurred  debt  in  buying  a  new  car  in  1956.  The  questionnaire 
was  short,  focusing  on  new  cars  and  on  new  car  debt,  and  lasted  on  the 
average  about  20  minutes.55  The  one  interviewer  employed  was  aware  of 
the  purposes  of  the  study  and  was  informed  of  the  respondent's  debt  sit- 
uation before  the  interview. 

The  second  Study  (S2b)  followed  up  on  the  previous  one  with  inter- 
views of  a  probability  sample  of  105  owners  of  new  car  debt  in  Chicago 
in  mid- 1957.  In  addition  to  seeking  further  information  on  the  nature  of 
response  error,  a  split-run  design  was  used  to  test  two  hypotheses:  ( 1 )  that 
response  error  would  depend  on  whether  or  not  the  interviewer  knew  the 
amount  of  the  loan  and  (2)  that  response  error  would  depend  on  the 
length  of  the  questionnaire. 

To  test  these  hypotheses,  the  sample  addresses  were  divided  by  random 
methods  into  four  equal  segments,  and  each  interviewer  was  given  an 
equal  number  of  interviews  in  each  segment.   These  segments  were : 

Group  Interviewer's  knowledge  Length  of  questionnaire 

1  Knew  amount  of  loan  Long  questionnaire 

2  Knew  amount  of  loan  Short  questionnaire 

3  Did  not  know  amount  Long  questionnaire 

4  Did  not  know  amount  Short  questionnaire 

Before  interviewing  sample  members  in  Groups  1  and  2,  interviewers 
were  told  the  amount  of  the  loan.  They  were  not  given  such  information 
in  interviewing  sample  members  in  Groups  3  and  4  (although  they  doubt- 
less were  aware  that  these  people  did  possess  car  debt) . 

The  questionnaire  used  on  Groups  2  and  4  was  considerably  shorter 
than  that  used  on  the  other  two  groups,  requiring  about  half  an  hour 
on  the  average  as  compared  with  almost  one  hour  for  the  longer  form. 
Like  the  shorter  questionnaire,  the  longer  form  focused  on  new  cars  and 
car  debt  but  sought  information  on  other  debts  as  well. 

The  third  of  these  three  studies  (S2c)  was  undertaken  concurrently 
with  the  others,  to  obtain  additional  data  on  response  errors  on  car  debt. 
It  consisted  of  searching  through  the  completed  questionnaires  of  the  1956 
Survey  of  Consumer  Finances  for  those  respondents  who  had  reported  a 
new  car  purchase.  For  such  names,  an  attempt  was  made  to  obtain  from 
the  title  office  in  the  appropriate  state  information  on  whether  or  not  there 
was  a  lien  on  the  purchase  and,  if  so,  the  amount. 

Offhand,  results  obtained  from  such  a  procedure  would  seem  of  limited 
applicability  because  respondents  who  had  not  reported  such  a  purchase 
could  not  be  included,  as  was  also  true  of  respondents  residing  in  states 
in  which  the  title  offices  were  unable  or  unwilling  to  supply  the  necessary 
information.    In  this  sense,  errors  obtained  from  such  a  check  might  be 

55  Lansing,  Ginsburg,  and  Braaten,  op.  cit.,  p.  21. 
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construed  as  lower  limits  to  the  true  amount  of  response  error  in  the  data. 
From  a  different  perspective,  however,  such  a  check  could  serve  to  throw 
greater  light  on  the  magnitude  of  response  errors  because  of  the  more 
realistic  circumstances  under  which  the  interviews  were  carried  out.  Thus, 
the  coverage  of  the  questionnaire  in  the  1956  Survey  of  Consumer  Fi- 
nances was  very  broad,  covering  all  debts  in  addition  to  new  car  debt  and 
also  covering  a  wide  range  of  assets  and  various  other  aspects  of  consumer 
saving  behavior.  The  interviewers  clearly  had  no  forewarning  of  such  a 
check,  because  it  was  not  even  contemplated  at  the  time  that  the  1956 
survey  was  conducted.  Unfortunately,  however,  this  check  produced  only 
33  comparisons,  limited  to  a  few  states. 

A  summary  of  the  extent  and  nature  of  response  errors  obtained  in 
these  three  studies  is  presented  in  Table  9.  In  some  respects,  the  results 
differed  considerably  from  one  study  to  another,  but  nevertheless  certain 
basic  characteristics  emerge.  Thus,  the  average  debt  of  the  nonreporters 
appears  to  be  generally  considerably  below  the  average  debt  of  those  who 
did  report.  Despite  this  fact,  however,  the  average  reported  debt  for  all 
the  respondents,  which  would  presumably  be  taken  as  the  estimate  of  the 
population  average  debt,  understates  the  average  actual  debt  of  all  re- 
spondents in  all  three  studies  (line  10),  substantially  the  same  as  in  the 
check  of  the  1956  Survey  of  Consumer  Finances.  The  principal  reason  for 
these  understatements  is  respondent  nonreporting  which,  as  shown  in  line 
11,  varied  from  less  than  4  percent  to  over  40  percent  of  the  total  inter- 
views in  these  three  studies. 

In  a  similar  fashion,  the  average  response  error  (for  respondents  who 
reported  debt)  varied  from  virtually  nothing  in  Study  S2a  to  almost  10 
percent  in  Study  S2c.  In  the  latter  case,  a  substantial  tendency  toward 
underreporting  is  evident  from  the  fact  that  more  than  one-third  of  the 
sample  members  understated  the  actual  debt  by  10  percent  or  more;  this 
error  served  to  contribute  to  the  large  overall  net  error  of  Study  S2c.  The 
same  tendency  toward  understatement  is  apparent  in  the  other  two  studies, 
even  though  the  average  error  in  Study  S2b  points  toward  overstatement 
(due  primarily  to  two  unusually  high  overestimates  of  the  actual  debt). 

The  allocations  of  error  in  the  last  four  lines  of  the  table  are  not  com- 
parable between  studies  because  the  nonrespondent  problem  was  not  con- 
sidered in  Studies  S2a  and  S2c.56   Nevertheless,  these  results  support  the 

50  It  was  not  considered  fully  in  Study  S2b  either,  because  no  data  were 
compiled  on  the  debt  of  those  sample  members  who  refused  or  were  not  contacted. 
The  nonrespondents  covered  in  Table  9  are  only  "item  nonrespondents,"  people 
who  were  interviewed  but  did  not  answer  (or  were  not  asked)  the  question  on 
car  debt.  The  allocation  of  error  for  Study  S2b  is,  therefore,  as  for  the  other 
studies,  also  incomplete.  Note,  however,  that  omission  of  the  noninterviews  does 
not  necessarily  imply  that  the  error  due  to  nonrespondents  is  underestimated.    The 
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Table  9.  Response  Errors  in  Surveys  of  Car  Debt,  1957, 
and  Cash  Loans,  1959 


Characteristic 


Survey  Research  Center  studies 


S2a 


S2b 


S2c 


S4a 


1.   Method  of  sample  selection. 


10. 
11. 


12. 
13. 

14. 
15. 


16. 
17. 
18. 


Sample  size,  initial 

Noncontacts 

Refusals 

Respondents 

Average  actual  debt  of  all  respondents 
Average  actual  debt  of  respondent 

nonreporters 

Average  actual  debt  of  respondent 

reporters 

Average  reported  debt  of  all  respon- 
dents (including  nonreporters) .  .  . 
Overall  net  error:  l-(line  9)/(line  6) 
Nonreporters :  percentage  of  re- 
spondents not  mentioning  debt. . . 
Reporters:    response  errors 

Average  percentage  of  error 

Percentage  overstating  by  10  per- 
cent or  more 

Percentage  within  10  percent 

Percentage  understating  by  10 

percent  or  more 

Allocation  of  error  by  percent 

To  nonrespondents 

To  nonreporting  of  respondents. . . . 
To  inaccurate  reporting  of 

respondents 


Judg- 
ment 
25 

c 

c 

25 
$2,075 

1,642 

2,090 

2,005 
3.5 

4.2 

.2 

4.2 
87.5 

8.3 

C 

94'8 
5.2 


Proba- 
bility 
105 
12 
2 
91 
$1,126 

553 

1,526 

1,057 
6.2 

40. 7d 

-7.7 

14. 8e 
66. 7e 

18. 5e 

-83.9 

562.3 

-378.4 


Judg- 
ment 
33 


33 
H  ,516 

1,428 

1,544 

1,059 
30.1 

24.2 

9.5 

0 
64.0 

36.0 

0 

75.8 
24.2 


Proba- 
bility 
133b 
27 
12 
94 
$23.12 

23.24 

$22.95 

9.53 
59.0 

60.6 

-2.1 

10.8 
81.1 

8.1 

c 

103^7 
-3.7 


a  Dollar  figures  are  weekly  payments  rather  than  amount  of  loan;  latter  was  not  validated. 

b  Excluding  5  respondents  no  longer  in  debt  at  time  of  the  interview. 

c  Not  applicable  or  data  not  available. 

d  Includes  6  instances  in  which  debt  identification  is  not  clear. 

e  Restricted  to  54  loans  for  which  loan  reported  appeared  to  be  the  same  as  that  validated. 

Source:  J.  B.  Lansing,  G.  P.  Ginsburg,  and  Kaisa  Braaten,  An  Investigation  of  Response 
Error,  Studies  in  Consumer  Savings,  No.  2  (Urbana:  University  of  Illinois  Bureau  of  Economic 
and  Business  Research,    1961),   Chs.   Ill  and  VI  and  unpublished  data  sheets. 


principal  rinding  obtained  from  the  Federal  Reserve  data  that  the  primary 
cause  of  error  was  nonreporting  by  respondents.  This  is  by  far  the  prin- 
cipal cause  of  the  underestimates  yielded  by  all  three  studies.  Particularly 
interesting  in  this  respect  is  Study  S2b,  where  nonreporting  would  have 
increased  the  error  far  more  substantially  than  was  actually  the  case,  had 
it  not  been  for  the  offsetting  overstatement  in  the  debts  that  were  reported 
by  the  respondents  and  in  the  estimate  of  the  average  debt  of  the  non- 
respondents. 

magnitude  of  this  error  depends  not  only  on  the  number  of  such  sample  members, 
but  also  on  the  difference  between  their  average  debt  and  that  of  the  respondents. 
In  this  case,  the  latter  was  likely  to  be  much  smaller  than  that  obtained  only  from 
the  item  nonresponses. 
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The  two  experiments  undertaken  in  Study  S2b  led  to  inconclusive  re- 
sults. Interviews  for  which  advance  knowledge  was  given  to  the  inter- 
viewers did  produce  information  somewhat  more  accurate  than  that  ob- 
tained in  other  cases,  but  the  results  were  not  statistically  significant. 
Similarly,  there  was  some  evidence  that  the  short  questionnaire  was  more 
successful  than  the  long  questionnaire,  but  again  the  results  were  not  sig- 
nificant. By  hindsight,  the  results  of  both  experiments  may  well  have  been 
confounded  by  interviewer  perception  of  the  purposes  of  the  study. 

A  more  important  finding  of  Study  S2b  was  the  substantial  differences 
obtained  in  the  reporting  of  primary  loans  versus  secondary  loans.  Thus, 
primary  loans  appear  to  have  been  reported  in  at  least  70  percent  of  the 
interviews  where  they  were  encountered,  whereas  the  same  was  true  of 
less  than  half  of  the  secondary  loans.  Considerable  confusion  seems  to 
have  been  encountered  in  obtaining  data  on  secondary  loans  because  they 
do  not  appear  to  have  been  distinguished  very  clearly  in  respondents' 
minds. 

Turning  to  the  effect  of  the  bias  in  these  three  studies  on  the  reliability 
of  the  estimates,  the  following  tabulation  shows  that  only  in  Study  S2c  is 
the  true  standard  error  of  the  mean  substantially  larger  than  the  standard 
error  computed  from  the  interview  data. 

S2a  S2b  S2c 

k  =  E/dy 43  .63  3.2 

r 11  .002  .24 

Apparent  standard  error  of  mean  [a y) ..  .    $163  $110  $145 

True  standard  error  of  mean  (y/MSE).   $1 54  $119  $486 

Error-free  standard  error  (o*) .  .  , $137  $  97  $117 

In  Study  S2c,  the  bias  reached  substantial  proportions  due  to  the  high 
incidence  of  nonreporting.  As  a  result,  the  apparent  standard  error  of  the 
mean  understates  the  true  figure  by  almost  70  percent.  In  addition,  the 
data  show  that  in  the  absence  of  response  error  the  standard  error  of 
the  mean  would  have  been  reduced  appreciably.  In  part  this  is  due  to  the 
positive  intercorrelation  between  the  errors  and  the  true  values,  although 
the  main  effect  is  the  relatively  large  size  of  the  response  errors,  partic- 
ularly in  Study  S2c. 

Cash  Loans 

As  part  of  the  Consumer  Savings  Project,  the  Survey  Research  Center 
undertook  in  1959  a  study  of  response  errors  in  reports  of  cash  loans.  In 
contrast  to  new  car  debt,  cash  loans  are  traditionally  one  of  the  most  dif- 
ficult to  ask  about  in  personal  interviews.  With  the  experience  gained  in 
the  preceding  studies  on  new  car  debt,  as  well  as  in  two  prior  studies  of 
savings  accounts  (covered  in  Chapter  IV),  it  was  felt  desirable  to  explore 
this  more  difficult  area  of  consumer  debt. 
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The  study  was  carried  out  by  selecting  a  probability  sample  of  known 
borrowers  in  two  urban  areas.  Contrary  to  the  procedure  in  Study  S2b, 
interviewers  were  not  told  of  the  real  purpose  of  the  study  and,  to  allay 
suspicion  still  further,  some  names  and  addresses  were  selected  from 
sources  other  than  the  available  list  of  cash  borrowers.  In  further  con- 
trast to  the  previous  debt  studies,  the  questionnaire  was  a  general  one, 
covering  a  variety  of  topics,  similar  to  that  used  in  the  annual  Survey  of 
Consumer  Finances.  Among  the  topics  included  were  attitudes  on  eco- 
nomic events,  housing  arrangements,  management  of  children's  education, 
saving  practices  and  attitudes,  and  financial  information  covering  liquid 
assets,  mortgage  payments,  and  installment  purchases  as  well  as  cash 
loans.  Psychological  questions  were  asked  to  probe  two  dimensions  of 
personality  —  social  conformity  and  "personal  effectiveness"  ■ —  felt  to  be 
related  to  reporting  of  the  loan,  as  did  indeed  turn  out  to  be  the  case. 

From  an  experimental  point  of  view,  the  sample  was  divided  into  three 
random  groups.  With  members  of  Group  A,  interviewers  were  instructed 
to  ask  for  the  respondent  by  name.  With  Group  B,  interviewers  were 
given  the  name  of  the  person  to  contact  but  were  requested  not  to  use 
the  name  during  the  interview.  With  Group  C,  interviewers  were  also  re- 
quested not  to  make  use  of  the  name  of  the  respondent.  In  addition,  they 
were  to  utilize  a  "sealed-envelope"  technique,  by  which  the  respondent 
was  to  be  offered  the  option  of  filling  out  the  financial  data  himself  on  a 
separate  sheet,  sealing  it  in  an  envelope  addressed  to  the  Survey  Research 
Center,  and  having  the  interviewer  drop  the  envelope  into  the  mail.  The 
interviewer  was  requested  in  these  interviews  to  show  the  respondent  an 
IBM  card  and  stress  the  anonymity  of  the  data  and  the  importance  of  the 
study  in  throwing  light  on  the  manner  in  which  people  handle  their  finan- 
cial affairs. 

Excluding  names  and  addresses  selected  to  camouflage  the  purpose  of 
the  study,  the  sample  consisted  of  138  households.  As  shown  in  Table  9, 
interviews  were  conducted  at  99,  or  73  percent,  of  these  households;57 
nevertheless,  the  base  for  the  study  of  response  error  was  94  completed 
interviews,  because  5  of  the  respondents  had  paid  off  their  debt  between 
the  time  the  sample  was  selected  and  the  date  of  the  interview. 

As  is  evident  from  Table  9,  the  overall  error  was  far  larger  than  on 
any  of  the  preceding  pilot  studies.    The  average  debt  reported  by  the  re- 

57  The  large  proportion  of  households  not  interviewed  was  due  primarily  to 
the  desire  of  the  researchers  to  obtain  only  100  completed  interviews  for  purposes 
of  analysis.  Although  it  might  have  been  more  desirable  to  have  covered  as  much 
of  the  sample  as  possible  (for  there  is  always  the  possibility  that  the  accuracy  of 
those  interviewed  toward  the  end  of  the  study  may  be  different  from  that  of  sample 
members  interviewed  earlier),  after  100  interviews  had  been  achieved,  no  further 
interviews  were  attempted.    Lansing,  Ginsburg,  and  Braaten,  op.  cit.,  Chapter  VI. 
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spondents  understates  the  actual  average  by  59  percent  (line  10).  Virtu- 
ally the  entire  error  resulted  from  the  fact  that  in  this  case  the  respon- 
dents who  did  not  report  the  known  loan  (over  60  percent)  outnumbered 
those  who  did.58  More  than  80  percent  of  the  nonreporters  did  report  other 
cash  loans,  thus  supporting  the  assertion  of  the  researchers  that  the  degree 
of  cooperation  was  much  better  than  would  be  inferred  just  from  glancing 
at  the  figure  for  the  percentage  of  nonreporters.  However,  these  other 
loans  could  not  be  identified  as  the  loan  under  study.59 

Loans  that  were  reported  coincided  well,  on  the  average,  with  the  true 
figure  (line  12),  although  it  should  be  borne  in  mind  that  accuracy  of 
report  served  as  a  basis  for  identification  of  the  loan,  so  that  these  two 
factors  may  not  be  independent.  There  was  also  no  tendency  for  bias  in 
these  errors  to  be  in  any  particular  direction. 

In  such  a  case,  with  error  due  to  nonrespondents  not  being  considered, 
the  allocation  of  error  in  lines  16-18  leads  to  the  foregone  conclusion  that 
just  about  all  the  error  is  the  result  of  respondent  nonreporting.  Although 
no  definitive  reason  can  be  given  for  this  high  incidence  of  nonreporting, 
aside  from  the  general  sensitivity  of  consumers  to  providing  information 
on  this  type  of  debt,  breakdown  of  the  nonreporting  by  various  character- 
istics helps  to  throw  some  light  on  the  question.  A  pronounced  and  statis- 
tically significant  tendency  toward  reporting  of  the  known  loan  was  ap- 
parent as  income  fell,  as  education  rose,  and  as  age  of  head  declined;  no 
relationship  was  found  for  the  other  two  socio-economic  characteristics 
studied,  namely,  occupation  of  head  and  sex  of  respondent. 

The  reason  for  higher  frequency  of  reporting  among  younger  people 
and  among  those  with  more  education  is  not  difficult  to  understand,  in 
view  of  the  general  tendency  of  these  groups  to  be  more  cooperative  on 
surveys  of  this  type.  The  better  reporting  among  the  lower-income  levels 
is  less  understandable.  However,  the  investigators  offer  a  plausible  reason 
for  this  phenomenon,  namely,  that  cash  borrowing  is  the  sort  of  debt  to 
which  more  stigma  is  likely  to  be  attached  at  higher-income  levels.  "For 
a  person  with  a  high  or  moderately  high  income  to  admit  that  he  had 
to  borrow  cash  to  meet  his  expenses  would  be  incongruent  with  his  per- 
ceived status  position."60    This  may  well  explain  why  only  11  percent  of 

58  This  abstracts  from  possible  errors  due  to  faulty  estimates  of  cash  loans  of 
nonrespondents,  actual  data  for  which  were  not  available. 

09  A  basic  difficulty  in  this  study  as  well  as  in  the  prior  validation  studies  was 
that  the  respondent  was  not  asked  to  indicate  the  name  of  the  institution,  the 
date  when  the  debt  was  taken  out,  or  in  whose  name  the  debt  was  registered.  As 
a  result,  identification  had  to  be  made  on  the  basis  of  various  bits  of  circumstantial 
evidence.  In  this  study,  four  such  items  were  used:  the  monthly  payment,  the 
number  of  payments,  the  purpose  of  the  debt,  and  the  type  of  lending  agency.  In 
view  of  this  lack  of  direct  identification,  mismatching  of  debts  was  much  more 
likely. 

00  Lansing,  Ginsburg,  and  Braaten,  op.  cit.,  p.  103.    Italics  in  original. 


Validation  Studies:   Debt  63 

the  families  with  incomes  of  $10,000  and  over  reported  the  known  loan 
as  compared  with  58  percent  of  those  with  incomes  under  $5,000. 

Several  personality  characteristics  and  attitudes  were  also  related  to 
debt  reporting.  People  who  scored  high  on  a  series  of  questions  on  the 
acceptability  of  using  credit  were  much  more  likely  to  report  the  loan 
than  others.  Furthermore,  people  with  a  middle  tendency  toward  social 
conformity  were  more  likely  to  report  the  debt  than  people  with  either 
high  or  low  conformity  scores.  As  the  investigators  suggest,  "It  may  be 
that  people  with  low  conformity  scores  are  not  susceptible  to  the  influence 
of  the  interviewer  and  do  not  respond  to  her  request  for  information.  At 
the  other  end  of  the  scale,  people  with  high  conformity  scores  may  be  un- 
willing to  admit  to  such  doubtful  behavior  as  borrowing  cash  from  a  small 
loan  company."61 

Moreover,  people  with  low  scores  on  personal  effectiveness  (essentially, 
ability  to  make  up  one's  mind  and  to  plan  ahead)  were  significantly  more 
likely  to  report  the  loan  than  people  with  high  scores.  An  explanation  of 
this  finding  is  that,  "People  with  high  scores  on  effectiveness  are  able  to 
resist  the  pressure  from  the  interviewer  to  report  their  financial  situation, 
but  those  with  low  scores  simply  do  as  the  interviewer  requests  because 
they  lack  the  self-confidence  to  do  otherwise."62 

Reporting  of  debt  was  also  much  higher  with  the  sealed-envelope  tech- 
nique than  with  the  method  used  for  either  of  the  other  two  experimental 
groups.  Where  the  sealed-envelope  technique  was  used,  only  3  percent  of 
30  individuals  reported  no  loans,  as  contrasted  to  16  percent  of  the  64 
individuals  in  Groups  A  and  B. 

Finally,  in  view  of  the  substantial  bias  in  the  estimate  of  the  average 
debt,  it  is  not  surprising  to  find  that  the  apparent  standard  error  of  the 
mean  computed  from  the  survey  data  understates  by  a  tremendous  margin 
the  true  standard  error.  The  relevant  figures  are : 

k 9.7 

r 30 

Apparent  standard  error  of  mean  (<jy) $  1 .  20 

True  standard  error  of  mean  (v  MSE) $1 1 .  60 

Error-free  standard  error  (ax) $      -60 

In  this  particular  case,  everything  acts  against  accurate  estimation  of 
the  standard  error.  The  sample  is  relatively  large  and  the  actual  amounts 
borrowed  are  clustered  within  a  fairly  narrow  range,  so  that  the  standard 
error  of  the  average  amount  borrowed  ($23.12)  is  very  low  —  barely  $.60. 
At  the  same  time,  the  interview  data  missed  the  true  mean  by  more  than 
50  percent,  with  the  result  that  e  is  very  large,  and  hence  k  rises  to  9.7. 

^  Ibid.,  p.  110. 
"Ibid.,  p.  114. 
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In  addition,  the  intercorrelation  is  positive,  .30.  The  overall  result  is 
that  both  the  estimates  and  the  standard  errors  of  the  estimates  bear  little 
relation  to  the  true  values.  Furthermore,  in  the  absence  of  response  error, 
the  standard  error  would  have  been  half  the  apparent  value  and  almost 
95  percent  below  the  true  value  for  these  observations. 

Farm  Debt 

In  1960  a  farm  panel  was  launched  as  part  of  the  Consumer  Savings 
Project,  with  financial  assistance  from  the  United  States  Department  of 
Agriculture.  The  initial  sample  consisted  of  377  names  and  addresses  of 
farm  owners  or  operators  in  a  well-to-do  area  plus  66  additional  names 
and  addresses  for  pretest  purposes.  Names  and  addresses  were  selected  by 
probability  methods  primarily  from  lists  of  farmers  in  debt  or  of  farmers 
with  demand  or  time  deposits.  The  interviewers,  however,  were  only  told 
that  statistical  probability  methods  were  used  in  sample  selection. 

Aside  from  methodological  aspects,  the  study  sought  to  throw  light 
on  the  asset  and  debt  position  of  farmers  and  on  how  farmers  handled 
their  finances.  The  questionnaires  used  on  this  study  were  therefore  very 
broad,  covering  all  assets  and  debts  in  addition  to  farm  income,  saving 
practices  and  attitudes,  and  a  variety  of  attitudinal  information  (see 
Appendix  A  for  copies  of  some  of  these  questionnaires) .  Information  was 
asked  about  individual  assets  and  debts,  including  names  of  institutions, 
dates,  and  members  owning,  so  that  identification  problems  could  be  kept 
to  a  minimum.63 

Two  controlled  experiments  were  carried  out  on  the  first  wave  of  inter- 
views.64 One  experiment  involved  using  alternative  advance  letters.  One 
was  a  straightforward  appeal  based  on  the  need  for  data  on  how  farmers 
handle  their  finances.  The  other  letter  took  the  same  general  approach 
but  in  addition  sought  to  involve  the  sample  member  further  by  seeking 
his  suggestions  and  aid  in  improving  the  data-collection  procedure.  To 
this  end,  sample  members  sent  this  letter  were  given  a  form  to  fill  out 
after  the  interview  in  which  they  recorded  their  impressions  of  various 
parts  of  the  interview  and  were  asked  for  suggestions  on  improving  the 
conduct  of  these  interviews. 

The  second  experiment  involved  the  use  of  two  different  questionnaire 
forms.    With  one  form,  a  "holdings  form,"  used  on  half  the  sample,  re- 

83  Such  information  is  virtually  a  necessity  anyway  in  a  panel  study,  if  track 
is  to  be  kept  of  changes  in  individual  holdings  over  time.  From  a  substantive 
point  of  view,  these  data  are  also  of  considerable  value  in  studying  the  influence 
of  location  of  financial  services  on  patronage. 

64  Other  experiments,  carried  out  on  later  waves,  will  be  discussed  in 
Chapter  VI. 
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spondents  were  asked  for  a  detailed  list  of  their  holdings  of  assets  and 
debts;  the  other  half  of  the  sample  was  interviewed  with  a  "change  form" 
and  asked  for  detailed  changes  in  assets  and  debts  since  the  previous  July 
1  (the  interviews  were  in  the  fall)  with  only  a  rough  indication  of  total 
holdings  of  different  assets  and  debts.  This  experiment  was  designed  to 
throw  further  light  on  the  possibility  raised  in  earlier  project  panel  oper- 
ations that  people  might  be  more  willing  to  divulge  their  full  financial 
position  after  having  been  given  a  chance  to  convince  themselves  of  the 
authenticity  of  the  study  and  of  the  trustworthiness  of  the  interviewer. 
Furthermore,  since  the  ultimate  aim  of  many  such  studies  is  to  estimate 
saving,  this  experiment  serves  as  a  basis  for  comparing  the  accuracy  of 
reported  change  with  that  of  computed  change. 

To  avoid  confounding  of  the  two  experiments,  the  sample  was  divided 
into  four  equal  random  segments  by  systematic  selection.  Each  segment 
was  accorded  a  different  treatment,  as  follows : 

( 1 )  Holdings  form,  straightforward  appeal. 

(2)  Holdings  form,  appeal  for  advice. 

(3)  Change  form,  straightforward  appeal. 

(4)  Change  form,  appeal  for  advice. 

Including  the  pretest  sample,  interviews  were  attempted  with  409 
farmers;  the  other  34  names  and  addresses  were  not  eligible  for  various 
reasons  —  deceased,  moved,  no  longer  farming,  and  so  forth.    The  results 

of  these  contacts  were : 

Number  in  Number  in 

pretest  sample  regular  sample 

Respondents 57  308 

Nonrespondents 9  35 

Refusals 8  33 

Noncontact /  2 

Total 66"  343~ 

In  other  words,  roughly  99  percent  of  the  sample  members  were  con- 
tacted, and  of  those  contacted,  about  90  percent  were  interviewed,  so  that 
the  overall  interview  rate  was  87  percent  for  the  pretest  and  90  percent 
for  the  regular  sample.  This  was  a  much  better  performance  than  was 
obtained  on  any  of  the  other  savings  studies,  panel  or  otherwise.65 

Debt  information  served  as  a  source  of  sample  selection  for  272  of  the 
409  eligible  sample  members.   Of  this  number,  133  were  to  be  interviewed 

60  It  might  be  noted  that  the  basis  for  analysis  in  this  study,  as  in  the  other 
project  panels,  was  the  "savings  unit."  In  this  study,  a  savings  unit  consisted  of 
the  farm  owner,  or  operator,  his  wife,  all  children  under  16,  and  other  family 
members  in  the  same  dwelling  earning  less  than  $600  per  year  or  having  savings 
of  less  than  $600  in  their  own  name. 
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Table  10.  Response  and  Nonresponse  Errors  in  Reports 
of  Farm  Debt  on  Wave  1  of  Study  P3 


Category 

(1) 


1.  Total  sample" 

2.  Respondents 

3.  Nonreporters 

4.  Reporters 

Nonrespondents 

5.  Noncontacts 

6.  Refusals 

Allocation  of  error 

7.  To  nonrespondents. 

8.  To  nonreporting  of 

respondents 

9.  To  inaccurate  re- 

porting of 
respondents 


Number 
of  sav- 
ings units 

(2) 


133 

120 

22 

98 


13 


Number 
of  debts 

(3) 


150 

137 

26 

111 


13 


Average  size  of 
individual  debt 

Actual 
(4) 

Reported 

(5) 

$6,897 
7,052 
3,885 
7,794 

$6,751 
6,751 

8,333 

5,267 

-88.1% 

461.5 

-273.4 

Percentage 
deviation, 
reported 

from  actual 

(6) 


2.1 

4.3 

6^9 


a  Restricted  to  those  interviewed  with  a  holdings  form  and  for  whom  debt  information  was 
a  primary  validation  source. 


with  a  holdings  form  and  139  with  a  change  form.   The  results  of  these 

attempts  were  as  follows: 

Holdings  form         Change  form 

Respondents 120  132 

Nonrespondents 13  7 

Refusals 13  6 

Noncontacts  and  other 0  1 

Total 133  139 

Because  of  the  different  data  validated,  meaningful  comparisons  of  the 
accuracies  of  report  obtained  on  these  two  forms  are  not  possible  (at  least 
not  on  the  basis  of  this  one  wave  of  interviews) .  The  results  of  the  val- 
idation are  therefore  presented  separately  for  each  form. 

Holdings  Form 

A  general  summary  of  the  accuracy  and  completeness  of  report  of 
debt  on  the  holdings  form  is  shown  in  Table  10.  As  is  evident  from  this 
table,  the  reported  average  debt  understates  the  actual  figure  by  about  2 
percent.  This  understatement  appears  to  be  the  result  of  three  different 
types  of  errors,  which  do  not  fully  offset  each  other,  namely : 

( 1 )  Almost  one-fifth  of  the  respondent  debts  were  not  reported  in  the 
interview.   These  debts,  however,  were  generally  small  ones. 

(2)  On  the  other  hand,  the  debts  of  the  nonrespondents  —  in  this  case 
all  the  nonrespondents  were  refusals  —  averaged  well  below  the  average 
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Table  11.  Distribution  of  Errors  in  Debt  Reports,  Wave  13 
Holdings  Form,  Study  P3 


R  report  relative 

Percent  of  total 

Average  debt  as 
reported  by 

to  /  report 

(1) 

Number 
of  debts 

(2) 

Dollar 
amount* 

(3) 

R 

(4) 

/ 

(5) 

Overstated  by  25  percent  or  more 

Overstated  by  11  to  24  percent 

Overstated  by  1 0  percent  or  less 

No  discrepancy 

10.8 
3.6 
8.1 
14.4 
31.5 
19.8 
11.7 

7.4 

3.5 

5.2 

17.1 

30.9 

7.9 

28.0 

$13,980 
5,375 
5,105 
9,168 
7,475 
8,529 
4,565 

$6,322 
4,650 
4,916 
9,168 

Understated  by  10  percent  or  less 

Understated  by  11  to  24  percent 

Understated  by  25  percent  or  more .  .  . 

7,558 
9,380 
7,863 

Total 

99.9 

100.0 

$8,333 

$7,794 

Base 

111 

$856,653 

»  Based  on  I  report. 

debt  of  the  respondents  (line  2  versus  line  6),  with  a  resulting  tendency 
for  such  debts  to  be  overstated  if  estimated  from  data  supplied  by  the 
respondents. 

(3)  A  further  reinforcing  tendency  is  evident  in  the  fact  that  the 
average  debt  of  the  respondent  reporters  slightly  exceeded  the  actual  figure 
(line  4). 

The  results  of  these  offsetting  tendencies  are  brought  together  in  the 
error  allocation  in  lines  7-9  of  Table  10.  This  allocation  shows  that  the 
entire  underestimate  is  due  to  respondent  nonreporting  and  that  the  extent 
of  the  underestimate  would  have  been  much  larger  had  it  not  been  for 
overestimates  both  on  the  part  of  the  respondent  reporters  and  in  the 
imputed  debt  estimates  of  the  nonreporters.66 

The  understatement  of  the  averages  is  also  reflected  in  the  distribution 
of  errors,  shown  in  Table  1 1 .  Oddly  enough,  large  errors  appear  to  occur 
with  more  or  less  equal  frequency  —  about  1 1  percent  of  the  debts  were 
overstated  by  25  percent  or  more  and  an  almost  equal  proportion  were 
understated  by  this  percentage.  However,  striking  differences  appear  when 
the  frequency  of  small  errors  of  different  magnitudes  are  examined.    Al- 

66  It  may  seem  somewhat  peculiar  that  the  inaccurate  reporting  of  the  respon- 
dents is  responsible  for  an  overestimate  of  273  percent  when  the  average  reported 
debt  of  this  group  was  less  than  7  percent  above  the  actual  figure.  The  explanation 
lies  in  the  large  size  of  this  group.  Representing  nearly  three-fourths  of  all  the 
known  debts  of  these  sample  members,  any  error,  even  a  relatively  small  one, 
affects  the  aggregate  substantially  when  summed  over  this  number  of  sample  mem- 
bers.  Also,  with  the  overall  bias  so  low,  relative  deviations  tend  to  be  magnified. 
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Table  12.   Percentage  Distribution  of  Farm  Debt,  by  Size,  Wave  1, 
Holdings  Form,  Study  P3 


Size  of  debt 

(1) 

Balances  reported  by 
both  R  and  / 

Debts  of 

non- 
reporters 

(4) 

All 
validated 
debts  of  R 

(5) 

Debts  of 

Report 
of/ 

(2) 

Report 
ofR 

(3) 

spondentsa 
(6) 

Under  $1 ,000 

2.7 
14.4 
27.0 
19.8 
10.8 
13.5 
7.2 
4.5 

3.6 

17.1 

22.5 

24.3 

9.0 

9.9 

8.1 

5.4 

19.2 

34.6 

19.2 

11.5 

7.7 

3.9 

3.9 

5.8 
18.3 
25.5 
18.3 
10.2 
11.7 
6.6 
3.6 

$1,000  to  $2,499 

$2,500  to  $4,999 

$5,000  to  $7,499 

$7,500  to  $9,999 

$10,000  to  $14,999 
$15,000  to  $24,999 
$25,000  and  over 

38.4 
23.0 
15.4 
7.7 
15.4 

Total 

99.9 

99.9 

100.0 

100.0 

99.9 

Base  number 

111 

111 

26 

137 

13 

a  In  this  case,  all  nonrespondents  were  refusals. 


most  one- third  of  the  debts  were  understated  by  less  than  10  percent, 
whereas  less  than  one-tenth  were  overstated  by  less  than  10  percent.  Simi- 
larly, almost  one-fifth  of  the  debts  were  understated  by  10  to  25  percent, 
whereas  debts  overstated  by  10  to  25  percent  represented  less  than  4  per- 
cent of  the  total. 

Much  the  same  phenomenon  exists  when  comparisons  are  made  in 
terms  of  dollar  amounts  rather  than  numbers  of  debts.  In  addition,  debts 
understated  by  25  percent  or  more  now  assume  greatly  increased  impor- 
tance, which  one  would  expect  by  definition. 

Table  1 1  also  brings  out  the  fact  that  roughly  half  of  the  debts  reported 
were  accurate  within  10  percent  and  almost  four-fifths  were  accurate 
within  25  percent.  In  terms  of  dollar  amounts,  however,  only  two-thirds 
of  the  total  were  accurate  within  25  percent,  with  more  than  one-fourth 
having  been  understated  by  more  than  this  percentage. 

Columns  4  and  5  of  this  table  suggest  that  the  understated  debts  tend 
to  be  higher  than  those  which  are  overstated.  For  example,  the  35  debts 
understated  by  10  percent  or  more  averaged  $8,817,  whereas  the  16  debts 
that  were  overstated  by  this  proportion  averaged  $5,904. 

The  results  of  these  various  errors  are  reflected  in  the  alternative  debt 
distributions  presented  in  Table  12.  The  debt  distribution  obtained  from 
the  interviews,  column  3,  appears  to  approximate  reasonably  closely  the 
actual  distribution  both  of  those  debts  (column  2)  and  of  all  known  debts, 
including  those  of  the  nonreporters  (column  5).    Nevertheless,  systematic 
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differences  do  appear.  Thus,  the  respondent  debt  distribution  tends  to 
overstate  the  frequency  of  large  debts  and,  relative  to  all  known  debts  of 
the  respondents,  tends  to  understate  the  frequency  of  small  debts.  As  is 
apparent  from  column  4,  this  is  due  to  the  heavy  concentration  of  debts 
of  the  nonreporters  at  the  lower  end  of  the  distribution.  Column  6  sug- 
gests that  this  understatement  of  the  frequency  of  small  debts  and  over- 
statement of  the  frequency  of  large  debts  would  have  been  even  more 
pronounced  had  the  debts  of  the  nonrespondents  been  added;  these  were 
small  debts  more  frequently  than  were  respondent  debts. 

Evaluation  of  the  effect  on  the  standard  error  of  the  mean  of  these 
errors  indicates,  somewhat  surprisingly,  that  the  true  standard  error  in 
this  case  is  overestimated  by  the  sample  observations,  as  follows: 

k 42 

r 26 

Apparent  standard  error  of  mean  (cr^) $722 

True  standard  error  of  mean  {\/ MSE) $783 

Error-free  standard  error  (ax) $571 

The  reason  for  this  appears  to  be  the  wide  range  of  variation  both  in 
actual  debts  and  in  reported  debts  which,  combined  with  the  relatively 
small  sample,  serves  to  produce  relatively  high  measures  of  sampling 
variation.  Thus,  (Tx  is  $571,  whereas  o~y  is  $722.  Since  the  degree  of 
understatement  is  rather  moderate  (Table  10),  the  value  of  k,  the  bias 
indicator,  is  not  high.  Nevertheless,  in  the  absence  of  response  error  the 
standard  error  would  have  been  reduced  roughly  20  percent  relative  to 
the  sample  estimate.  In  contrast  to  previous  studies,  the  primary  reason 
for  this  is  the  positive  intercorrelation  between  the  actual  values  and  the 
response  errors;  this  term  accounts  for  two-thirds  of  the  reduction. 

Change  Form 

The  debt  information  on  the  change  form  seems  to  be  less  reliable  than 
on  the  holdings  form  although,  paradoxically,  the  change  form  appears  to 
have  been  more  successful  in  obtaining  interviews.  Thus,  of  those  con- 
tacted to  be  interviewed  with  a  change  form,  little  more  than  4  percent 
refused  (6  of  138),  as  shown  in  column  2  of  Table  13.  In  contrast,  13  of 
133  sample  members  contacted  with  the  holdings  form  refused  to  be  inter- 
viewed (see  Table  10,  p.  66),  a  refusal  rate  of  almost  10  percent.  The 
difference  is  only  statistically  significant  at  the  .08  probability  level  but, 
combined  with  information  to  be  presented  later,  suggests  that  the  change 
form  does  produce  a  higher  rate  of  response. 

Offhand,  such  an  inference  may  seem  strange,  since  the  respondent 
presumably  does  not  know  in  advance  what  type  of  questions  will  be  asked. 
The  explanation  lies  in  the  interchange  between  the  interviewer  and  the 
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Table  13.   Response  and  Nonresponse  Errors  in  Reports  of 
Change  in  Farm  Debt  on  Initial  Interview,  Study  P3 


Category 

Number 

of  savings 

units 

(2) 

Number 
of  debts 

(3) 

Average  change  per 
debt 

Percentage 
of  deviation, 

reported 
from  actual 

(6) 

(1) 

Actual 

(4) 

Reported 

(5) 

Total  sample" 

Respondents 

Nonreporters 

Reporters 

Nonrespondents 

Noncontacts 

Refusals 

139 

132 

55 

77 

7 

1 

6 

147 

140 

57 

83 

1 

1 

6 

S-62 

-31 

-78 

-1 

667 

0 

778 

884 
88 

148 

135.5 
283.9 

14,800 ! 6 

"  Restricted  to  those  interviewed  with  a  change  form  and  for  whom  debt  information  was  a 
primary  validation  source. 


respondent  before  the  interview  begins.  With  reluctant  respondents,  the 
interviewer  often  has  to  divulge  much  more  information  about  the  inter- 
view than  is  otherwise  necessary.  If  this  discussion  kept  on  long  enough 
the  interviewer  did  indicate,  according  to  reports,  the  type  of  questions 
to  be  asked  by  that  approach.  It  was  clearly  much  easier  to  use  the  change 
form  as  an  inducement  to  grant  an  interview  than  to  do  this  with  the 
holdings  farm,  since  in  the  former  case  the  interviewer  could,  and  did, 
stress  the  fact  that  only  changes  had  to  be  reported  and  not  aggregate 
holdings.67 

The  results  of  this  interchange  apparently  served,  oddly  enough,  to 
increase  the  reliability  of  the  data  obtained  on  the  holdings  form.  The 
reason  appears  to  have  been  the  greater  tendency  of  the  less  cooperative 
sample  members  to  drop  out  when  approached  with  a  holdings  form  than 
was  their  tendency  when  approached  with  a  change  form.  Thus,  almost 
42  percent  of  the  respondents  on  the  change  form  failed  to  report  one  or 
more  debts  (Table  13) ,  whereas  the  corresponding  percentage  on  the  hold- 
ings form  was  only  18  percent.  In  addition,  on  the  change  form  the 
number  of  debts  reported  per  respondent  averaged  1.04,  whereas  on  the 
holdings  form  the  number  of  debts  reported  per  respondent  averaged  1.14. 
Since  the  two  subsamples  are  presumably  random,  this  would  suggest  that 
more  respondents  report  debt  on  the  holdings  form  than  on  the  change 
form  and,  in  addition,  that  more  complete  information  is  obtained  even 
for  the  reporters  on  the  holdings  form. 

Columns  4-6  of  Table  13  also  suggest  that  the  quality  of  the  debt  infor- 

07  In  addition,  on  a  few  interviews  a  refusal  was  encountered  after  the  inter- 
view was  begun.  These  interviews,  which  were  treated  as  refusals,  were  nearly 
always  encountered  with  the  holdings  approach. 
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Table  14.  Distribution  of  Errors  in  Debt  Reports,  Wave  1. 
Change  Form,  Study  P3 


Increases  in  debt 

Decreases  in  debt 

No 
change: 
percent- 
age of 
debts 

R  report  of  change 
relative  to  /  report" 

Percent- 
age of 
debts 

Percentage 
of  dollar 
change 

Percent- 
age of 
debts 

Percentage 
of  dollar 
change 

Overstated  by  25  percent 

or  more 

Overstated  by  11  to  24  percent 
Overstated  by  10  percent 

or  less 

No  discrepancy 

Understated  by  10  percent 

or  less 

Understated  by  1 1  to  24  percent 
Understated  by  25  percent 

or  more 

33.3 

11.1 
55.6 

65.1 

6'8 
28.0 

25.9 

11.1 

14.8 
7.4 

3.7 
37  A 

18.4 
40.9 

14.3 
6.1 

7.3 
13.0 

5.3 
94.7 

Total 

100.0 

99.9 

100.0 

100.0 

100.0 

Base  number 

18 

$27,026 

27 

$27,078 

38 

a  Overstatements  and  understatements  refer  to  absolute  changes. 


mation  reported  on  the  change  form  may  not  be  high.  The  interview  re- 
ports indicate  that  the  debt  of  the  respondents  increased  during  the  three 
to  four  months  between  July  1  and  the  date  of  the  interview  by  about  $88, 
on  the  average.  Actually,  these  debts  decreased  by  $31  on  the  average 
and  by  $62  for  the  entire  subsample.  As  a  result,  the  direction  of  change 
was  missed  completely.  Contrary  to  the  situation  with  the  holdings  form, 
the  error  appears  to  have  been  equally  characteristic  of  reporters;  indeed, 
it  was  the  largest  of  all:  they  reported  an  average  debt  increase  of  $148, 
whereas  the  amount  of  debt  actually  declined  by  $1. 

Reasons  for  these  errors  become  evident  in  Table  14.  This  table  shows 
the  following: 

( 1 )  Paradoxically,  of  debts  that  increased  in  amount,  more  than  half 
were  understated  by  25  percent  or  more,  although  one-third  were  over- 
stated by  25  percent  or  more.  The  latter  group  accounted  for  almost  two- 
thirds  of  the  dollar  increase  in  these  debts. 

(2)  The  absolute  change  of  most  debts  that  decreased  in  amount  was 
overstated.  This  was  true  both  of  more  than  half  of  the  number  of  such 
debts  and  of  almost  three-fourths  of  the  dollar  change.  The  decrease  of 
more  than  one-third  of  these  debts  was  understated  by  25  percent  or  more, 
although  representing  only  13  percent  of  the  total  dollar  amount. 

(3)  Debts  which  had  not  changed  in  amount  were  as  a  rule  correctly 
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Table  15.   Percentage  Distribution  of  Change  in  Farm  Debt, 
by  Size,,  Wave  1,  Study  P3 


Amount  of  change 

(1) 

Changes 
by  both 

Report 
of/ 

(2) 

reported 
R  and  / 

Report 
of/? 

(3) 

Changes 
not  re- 
ported by  R 

(4) 

All 
validated 
changes 

(5) 

—  $1,500  and  under 

6.1 
6.1 
4.9 
12.2 
3.7 

46.3 

1.2 
3.7 
8.5 
3.7 
3.6 

8.5 
7.3 
1.2 
6.1 
0 

59.8 

0 
3.7 
4.9 
2.4 
6.1 

10.7 

14.3 
8.9 
1.8 

44.6 

5  A 
8.9 
3.6 
1.8 

6.5 

-$1,000  to  -$1,499 

-$500  to  -$999 

3.6 
8.6 

-$100  to  -$499 

-$1  to  -$99 

10.8 
3.6 

$1  to  $99 

45.3 

0.7 

$100  to  $499 

4.3 

$500  to  $999 

9.4 

$1,000  to  $1,499 

$1,500  and  over 

3.6 
2.8 

Total 

100.0 

100.0 

100.0 

99.8 

Base  number 

83 

83 

56 

139 

reported.   For  only  two  of  38  such  debts  were  errors  made,  both  of  which 
were  overstatements. 

The  reason  for  this  substantial  overstatement  in  the  aggregate  change 
in  debt  now  becomes  clear.  According  to  the  institutional  data  (/) ,  the 
aggregate  decrease  in  debt  was  only  slightly  larger  than  the  aggregate 
increase.  However,  debts  that  increased  were  substantially  overstated, 
whereas  debts  that  decreased  were,  on  the  average,  only  slightly  overstated 
(in  absolute  amount).  The  figures  are  as  follows,  with  R  denoting  re- 
spondent reports: 

Increase  Decrease 

I $27,026     $27,078 

R 41,670     29,379 

Largely  as  a  result  of  these  overstatements,  the  net  reported  increase  in 
debt  was  $12,291  instead  of  the  actual  figure  of  —$52. 

The  effect  of  these  errors  on  the  distribution  of  debts  by  amount  of 
change  is  shown  in  Table  15.  Perhaps  most  surprising  about  this  table  is 
that  despite  the  large  error  in  the  aggregate,  the  reported  distribution  of 
change  (column  3)  is  not  too  different  from  the  actual  distribution,  either 
for  respondents  alone  or  for  all  validated  changes.  Thus,  the  R  distri- 
bution indicates  that  about  17  percent  of  the  debts  increased  in  amount, 
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whereas  the  corresponding  true  figure  was  19.5  percent.  According  to  the 
R  distribution,  23  percent  of  the  debts  decreased  by  $100  or  more,  whereas 
the  true  figure  was  29  percent. 

The  most  pronounced  errors  were  in  the  tendencies  of  the  reporters 
to  overstate  the  frequency  of  no  change,  as  well  as  the  frequency  of  change 
of  more  than  $1,000  at  both  ends  of  the  distribution.  Since  changes  in 
debt  not  reported  appear  to  have  involved  more  frequently  amounts  of 
less  than  $1,000  (column  4),  the  distribution  of  reported  change  tends 
to  overstate  even  further  the  relative  frequency  of  large  changes. 

Determinants  of  Error 

From  the  foregoing  it  is  abundantly  clear  that  reporting  of  debt  was 
affected  by  type  of  questionnaire  form.  Indeed,  both  of  the  experiments 
appear  to  have  had  some  effect  on  debt  reporting,  although  only  the  joint 
effect  of  the  two  is  statistically  significant  with  regard  to  frequency  of 
debt  reported.  The  relevant  data  are  as  follows: 


Percentage  of 

Base 

saving 

units 

(number  of 

Letter 

Form 

reporting  debt 

savings  units) 

Regular 

Holdings 

73.8 

84 

Change 

76.0 

96 

Total 

75.0 

180 

Evaluation 

Holdings 

88.6 

88 

Change 

77.9 

95 

Total 

83.1 

183 

Total 

Holdings 

81.4 

172 

Change 

77.0 

191 

Total 

79.1 

363 

More  frequent  reports  of  debt  —  and,  hence,  presumably  more  com- 
plete data  —  apparently  occurred  on  the  holdings  form  than  on  the 
change  form,  and  this  was  also  true  when  the  evaluation  letter  (asking  for 
advice  and  suggestions)  was  used  in  place  of  the  regular  letter;  the  latter 
difference  —  between  83.1  percent  and  75.0  percent  —  is  barely  not  statis- 
tically significant  at  the  .05  probability  level.  The  highest  frequency  of 
debt  reports  by  far,  almost  90  percent,  was  achieved  with  the  combined 
use  of  the  holdings  form  and  the  evaluation  letter.  This  percentage  is 
statistically  above  (at  the  .05  significance  level)  the  percentages  for  each 
of  the  other  three  combinations.  Evidently,  the  interaction  of  these  two 
factors  yields  results  superior  to  either  one  alone. 

Results  of  tests  of  the  effects  on  reporting  of  farm  debt  of  a  number 
of  respondent  interviewer  and  interview  characteristics  are  summarized 
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Table  16.   Chi-Square  Tests  of  Effect  of  Different  Interview 
Characteristics  on  Reporting  of  Farm  Debt,  Study  P3 


Characteristic 

Value  of 
chi-square 

Degrees  of 
freedom 

Significance 
level 

Age  of  farm  operator 

Education  of  farm  operator 

5.020 
4.044 
18.706 
15.356 
14.986 
.933 
32 . 347 

9.593 

3.891 

8.292 

.910 

15.347 

7.411 

20.782 

18.256 

8 
8 

12 
12 
14 
2 
14 

6 
6 
4 
2 
4 
6 
12 
10 

.76 
.85 
.10 

Value  of  total  assets 

.22 
.45 

Renter  versus  owner 

Size  of  debt 

.6 
.004 

Interviewer  ratings  of: 

.15 

Completeness  of  financial  data 

Accuracy  of  debt  reporta 

Completeness  of  debt  report 

Use  of  records  on  checking  accounts .... 
Interviewer  rating  when  hired 

.69 
.15 
.63 

.01 
.24 

Interviewer  dominance  score  on  EPPSb.  . 
Interviewer  deference  score  on  EPPS .... 

.06 

.05 

a  Chi-square  is  6.012  with  2d.f.  (P  =  .05)  if  test  is  carried  out  between  reporters  and  non- 
reporters. 

b  Edwards  Personal  Preference  Schedule,  discussed  in  Mathew  Hauck  and  Stanley  Steinkamp, 
Survey  Reliability  and  Interviewer  Competence  (Urbana:  University  of  Illinois  Bureau  of  Economic 
and  Business  Research,  1964),  pp.  39-44. 


in  Table  16.  The  chi-squares  in  this  table  were  obtained  by  classifying 
the  validated  debts  into  three  categories:  those  which  were  not  reported, 
those  which  were  reported  within  25  percent  of  the  actual  figure,  and 
those  which  were  reported  but  deviated  by  more  than  25  percent  from 
the  actual  figure.68  This  classification  differs  from  the  simple  reporter- 
nonreporter  distinction  used  previously  but,  with  one  exception  to  be 
mentioned  shortly,  yields  the  same  results  in  either  case. 

Although  most  of  the  characteristics  in  this  table  are  not  significant 
at  the  .05  probability  level,  four  are  significant  or  nearly  so,  and  several 
others  would  seem  to  merit  further  consideration.  Of  the  character- 
istics which  are  significant,  one  refers  to  a  feature  of  the  debt  itself, 
namely,  its  size ;  one  refers  to  a  measure  of  respondent  cooperativeness  — 
use  of  records  on  checking  accounts;  and  two,  rather  surprisingly,  refer  to 
personality  characteristics  of  the  interviewer,  as  based  on  scores  obtained 
from  the  Edwards  Personal  Preference  Schedule.  (EPPS)  test. 

In  each  case,  the  significance  of  the  relationship  is  supported  by  its 
plausible  nature.  Thus,  those  with  large  debts  were  less  likely  to  report 
their  existence  than  those  with  small  debts.  People  who  used  records  in 
giving  checking  account  balances  were  more  likely  to  report  the  debt  ac- 


There  were  no  instances  in  which  the  debt  was  reported  but  the  balance  was 


refused. 
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curately,  and  they  were  even  more  likely  still  to  report  the  existence  of 
the  debt.69 

Interviewers  with  low  scores  on  dominance  were  much  more  likely 
than  those  with  high  scores  to  turn  in  questionnaires  with  no  debt  re- 
ported, as  was  also  true  of  interviewers  with  high  scores  on  deference.  To 
some  extent,  one  of  these  characteristics  is  the  complement  of  the  other, 
but  there  is  some  indication  that  taken  together  they  provide  more  infor- 
mation than  each  one  separately. 

Table  16  indicates  further  that  the  interviewer  may  be  able  to  gauge 
the  accuracy  of  a  report,  and  that  accuracy  of  report  may  be  a  more  sensi- 
tive indicator  from  the  interviewer  standpoint  than  completeness  of  report. 
In  particular,  a  chi-square  test  of  the  interviewer's  estimate  of  accuracy 
of  report  of  the  debt  related  to  reporting  versus  nonreporting  was  sig- 
nificant at  the  .05  probability  level. 

It  is  interesting  to  note  that  none  of  the  socioeconomic  characteristics 
tested  was  statistically  significant,  not  even  age  which  had  been  significant 
in  many  previous  studies.  There  is  some  indication,  however,  that  those 
at  the  very  lowest  income  levels  were  likely  to  be  more  inaccurate  and  also 
more  likely  not  to  report  the  debt  than  those  at  higher  income  levels. 

Two  Secondary  Validation  Studies 

After  the  sample  had  been  selected  in  Studies  PI  and  P2,  a  limited 
amount  of  debt  validation  by  obtaining  debt  information  from  a  central 
credit  source  in  each  area  proved  feasible.  Such  secondary  validation  can- 
not possess  the  reliability  of  primary  validation  for  various  reasons,  two  of 
them  general  and  others  specific  to  the  particular  sources  used.  Perhaps 
most  important  of  the  general  reasons  is  the  fact  that  no  debt  information 
is  obtained  for  the  nonrespondents,  with  some  exceptions  to  be  noted  later. 
Second,  a  nonreported  holding  is  less  likely  to  be  picked  up  by  the  credit 
source  than  a  reported  one ;  it  is  clearly  impossible  for  any  one  institutional 
source  to  cover  all  forms  of  debt,  including  debt  to  individuals  and  to 
neighborhood  merchants. 

The  specific  reasons  relate  to  the  validation  source  used,  which  was  the 
same  in  both  of  these  studies.  Since  the  source  operated  as  a  clearing- 
house for  member  institutions,  its  data  depended  on  the  scope  and  co- 
operativeness  of  these  institutions  and  on  the  care  taken  in  supplying  the 
data.    In  one  of  these  studies  (PI),  many  of  the  institutions  in  the  area, 

69  The  reason  for  testing  use  of  records  on  checking  accounts  rather  than  on 
some  other  asset  was  that  virtually  all  the  respondents  had  checking  accounts  so 
that  classification  by  this  asset  yielded  information  for  most  of  the  sample  members. 
Use  of  records  on  debt  was  out  of  the  question  because  if  a  respondent  had  no 
debt  or  did  not  report  it,  records  yielded  nothing. 
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including  most  of  the  nonretail-debt-handling  institutions,  were  not  part 
of  this  clearinghouse  operation.  In  addition,  some  of  the  member  institu- 
tions in  both  areas  exhibited  erratic  reporting  practices,  tending  to  with- 
hold debt  information  on  "good  customers."  Moreover,  the  procedure 
used  by  these  central  sources  to  update  debt  information  was  to  ask  for 
such  information  from  previously  reported  creditors  only.  Presumably, 
the  central  file  would  contain  for  each  individual  a  complete  list  of  his 
creditors  insofar  as  prospective  creditors  asked  for  previous  debt  infor- 
mation before  extending  credit,  and  this  request  was  recorded  in  the 
individual's  file  for  later  follow-up.  However,  if  a  prospective  creditor 
did  not  ask  for  such  information,  perhaps  not  feeling  any  need  for  debt 
information  on  the  prospective  borrower,  the  central  clearing  source  would 
not  be  aware  of  the  new  creditor.  Although  creditors  were  supposed  to 
report  all  new  loans  or  credit,  circumstantial  evidence  indicates  they  did 
not  always  do  so. 

A  final  limitation  of  these  data  is  the  difficulty  of  matching  debts 
and  debt  balances,  with  regard  both  to  identity  and  to  time  of  a  given 
balance.  The  latter  is  invariably  reported  by  the  creditor  as  of  the  date 
of  the  request.  Since  the  requests  had  to  be  sent  out  some  time  after  the 
interviews  had  been  completed  —  generally  several  weeks  —  it  was  often 
not  easy  to  extrapolate  a  debt  back  to  the  time  of  the  interview,  even 
though  the  terms  of  the  debt  were  generally  stated.  Also,  with  retail  debt 
a  matching  problem  frequently  arose,  since  the  one  balance  figure  could 
represent  several  debts  as  well  as  debts  by  more  than  one  family  member. 
The  existence  of  the  latter  could  only  be  discerned  from  the  interview  data. 

For  these  reasons,  the  results  of  the  secondary  validation  studies  are 
subject  to  errors  not  present  in  the  other  studies  and  are  not  strictly  com- 
parable with  the  other  studies.  The  table  forms  and  presentation  of  results 
in  this  section  reflect  this  fact  and  differ  from  those  of  the  previous  sections 
in  various  ways,  as  will  be  noted  shortly. 

The  two  secondary  validation  studies  were  carried  out  under  similar 
conditions,  and  both  purport  to  cover  all  forms  of  nonmortgage  debt. 
However,  since  there  are  important  differences  in  the  manner  in  which 
each  was  conducted,  as  well  as  in  coverage  of  debt  by  each  of  the  two 
validation  sources,  it  seems  preferable  to  present  the  results  of  these  studies 
separately,  making  comparisons  whenever  they  are  appropriate. 

Study  PI 

This,  the  first  panel  validation  study,  was  concerned  with  secondary 
validation  of  nonmortgage  debt  as  well  as  of  life  insurance.  In  addition, 
the  operation  was  designed  to  test  the  effectiveness  of  various  interview 
approaches  toward  soliciting  financial  data.   To  this  end,  three  controlled 
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experiments  were  incorporated  into  the  first  of  the  five  waves  of  interviews 
in  this  operation.  These  experiments  were : 

(1)  Use  of  a  relatively  brief,  vague  advance  letter  on  the  purposes 
of  the  study  versus  a  longer,  more  detailed  advance  letter.  (For  copies, 
see  Appendix  A.) 

(2)  Dollar  figures  requested  on  the  first  wave  versus  no  dollar  figures 
requested  until  the  second  wave.  In  the  latter  case,  only  general  infor- 
mation on  asset  and  debt  holdings  was  sought  on  the  first  wave,  giving 
the  respondent  and  interviewer  a  chance  to  get  to  know  each  other  before 
taking  up  dollar  amounts. 

(3)  Use  of  a  structured  questionnaire  with  the  interviewer  required 
to  follow  strictly  the  order  and  wording  of  the  questions  versus  an  un- 
structured approach  whereby  the  interviewer  asked  questions  from  memory 
in  whatever  order  seemed  most  suitable,  jotted  notes  on  a  pad  during  the 
interview,  and  wrote  a  detailed  report  immediately  afterward.  The  pur- 
pose of  this  latter,  depth-type  interview  situation  was  to  ascertain  whether 
or  not  it  left  the  respondent  (and  the  interviewer)  more  at  ease  and  hence 
presumably  more  likely  to  discuss  finances  frankly. 

As  a  result  of  these  experiments,  four  different  questioning  approaches 
were  used  on  the  first  wave  of  interviews;  copies  of  the  two  structured 
questionnaires  are  shown  in  Appendix  A.  To  avoid  the  confounding  of 
one  experiment  with  another,  the  sample  was  divided  by  systematic  selec- 
tion into  eight  random  segments,  each  segment  receiving  a  different  com- 
bination of  these  experimental  approaches  in  a  manner  analogous  to  that 
described  for  Study  P3  (p.  65) .  In  all  instances,  the  questionnaire  covered 
the  entire  spectrum  of  consumer  assets  and  debts,  including  annuities, 
pensions,  brokerage  accounts,  and  ownership  of  a  business.  Where  dollar 
figures  were  requested,  the  subject  was  always  holdings,  not  the  change 
since  a  particular  past  period. 

The  sample  for  Study  PI  was  selected  by  area  probability  methods 
from  the  population  of  a  large  urban  area,  using  1950  census  tract  data. 
Disproportionate  representation  was  given  to  the  high-income  areas  — 
census  tracts  in  which  25  percent  or  more  of  the  consumer  units  reported 
incomes  in  excess  of  $10,000  in  1949.  As  a  result,  over  40  percent  of  the 
sample  reported  incomes  above  $10,000  in  1958;  a  rough  estimate  of  the 
actual  proportion  of  such  consumer  units  in  the  area  at  that  time  is  10 
percent. 

The  initial  sample  for  this  study  comprised  308  names  and  addresses, 
selected  with  a  view  toward  securing  at  least  200  completed  interviews  on 
the  first  wave.  The  eligible  sample  turned  out  to  be  296,  because  of  at- 
trition due  primarily  to  deaths  and  migration  out  of  the  area.  Of  this 
number,  210  were  interviewed,  61  refused,  and  25  could  not  be  contacted. 
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In  other  words,  the  response  rate  (interviews/contacts)  was  78  percent; 
the  contact  rate  (contacts/sample)  was  92  percent;  and  the  overall  inter- 
view rate  was  7 1  percent. 

The  results  of  the  debt  validation  are  summarized  in  Table  17.  The 
principal  finding  evident  from  this  table  is  the  rather  startling  one  that  in 
no  instance  could  a  debt  reported  by  R  be  identified  as  also  having  been 
reported  by  I.  In  other  words,  the  debts  obtained  from  these  two  sources 
were  mutually  exclusive,  all  the  information  obtained  from  one  source 
serving  to  supplement  rather  than  to  verify  information  obtained  from  the 
other  source. 

There  is  reason  to  believe  that  this  state  of  affairs  was  a  peculiarity  of 
two  conditions  under  which  the  debt  information  was  obtained  from  the 
two  sources.  One  condition  was  the  unclear  manner  in  which  this  infor- 
mation was  sought  in  the  interview.  The  opening  question  on  debt  ad- 
vised the  respondent,  among  other  things,  not  to  include  "bills  payable  in 
30  days."  The  intent  of  this  phrase  was  to  exclude  debt  of  less  than 
30-days'  duration.  However,  because  of  the  manner  in  which  this  request 
was  worded,  respondents  were  encouraged  to  omit  any  debt  which  they 
thought  might  be  payable  within  30  days,  irrespective  of  the  length  of  time 
which  they  might  already  have  had  it.  By  hindsight,  a  considerable  num- 
ber of  retail  debts  may  fall  into  this  category,  since  people  often  expect  to 
pay  off  such  debts  within  the  next  30  days.  Second,  the  institutional  source 
appears  to  have  had  poor  cooperation  from  nonretail  creditors.  As  a 
result,  debt  obtained  from  this  source  was  almost  exclusively  retail  debt. 

These  circumstances  are  reflected  in  Table  17  in  a  much  larger  num- 
ber, but  much  lower  average  size,  of  debts  obtained  from  /  than  from  R. 
This  situation  is  reflected  even  more  strikingly  by  segregating  retail  debt 
from  nonretail  debt.  Doing  so,  we  find  that  for  Questionnaire  A,  on  which 
dollar  amounts  were  requested,  the  respondent  reports  represented  about 
90  percent  of  both  the  number  and  the  amount  of  nonretail  debt  obtained 
from  both  sources.  On  the  other  hand,  virtually  all  (95  percent)  of  the 
retail  debt  was  reported  by  /.  Moreover,  nonretail  debt  averaged  more 
than  15  times  higher  than  retail  debt.  As  a  result,  even  though  the  num- 
ber of  nonretail  debts  obtained  from  both  sources  for  the  respondents  was 
not  much  more  than  half  the  number  of  retail  debts,  nonretail  debts  rep- 
resented more  than  85  percent  of  the  total  known  debts  of  these  sample 
members.  It  is  this  difference  in  the  frequency  of  debt  reported  in  the 
average  size  of  debt  that  largely  accounts  for  the  mixed  picture  shown  in 
Table  17. 

Results  obtained  from  the  use  of  Questionnaire  Form  B,  in  which  no 
dollar  amounts  were  requested,  correspond  closely  with  those  from  Ques- 
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tionnaire  A.  There  are  some  differences  in  the  corresponding  average 
debts  obtained  by  the  two  forms,  based  on  the  I  reports,  but  these  differ- 
ences appear  to  be  largely  a  manifestation  of  the  small  sample  sizes.  In  the 
one  instance  in  which  the  sample  sizes  are  appreciable  —  the  reports  of 
retail  debt  by  /  only  —  the  two  averages  correspond  closely.  Indeed,  there 
appears  to  have  been  little  difference  in  the  reporting  of  this  debt  whether 
or  not  requests  were  made  for  dollar  amounts.  The  number  of  debts  ob- 
tained on  the  two  forms  was  much  the  same  in  either  case,  both  for  retail 
debt  and  for  nonretail  debt. 

Because  of  the  relatively  small  sample  sizes,  these  data  do  not  provide 
much  evidence  as  to  whether  or  not  the  debts  of  the  refusals  and  of  the 
noncontacts  are  any  different  in  size  or  any  more  or  less  frequent  than 
those  of  the  respondents.  With  regard  to  size,  the  variability  in  debts  re- 
ported for  refusals  and  noncontacts  is  too  high  to  permit  any  clear  infer- 
ences. With  regard  to  frequency,  however,  there  is  evidence  that  the 
refusals  and  noncontacts  might  have  been  more  often  in  debt  than  the 
respondents.  Debts  were  reported  by  I  for  35  percent  of  the  latter,  but  for 
57  percent  of  23  noncontacts  and  for  59  percent  of  49  refusals  for  which 
debt  reports  were  obtained.  To  be  sure,  most  of  these  debts  were  retail 
debts,  and  it  is  not  clear  whether  the  same  phenomenon  also  exists  for 
other  types  of  debt. 

Analysis  of  the  frequency  of  debt  report  by  the  other  two  experimental 
variables  indicates  that  no  appreciable,  nor  statistically  significant,  differ- 
ence is  obtained  whether  a  brief,  vague  advance  letter  is  used  or  a  longer, 
more  detailed  letter.  However,  the  structured  questionnaire  did  produce 
a  noticeably  higher  frequency  of  debt  reports  than  the  unstructured  ques- 
tionnaire—  31  percent  versus  19  percent,  respectively.  This  difference  is 
almost  statistically  significant  at  the  .05  probability  level  (/?  =  .065). 
More  important,  it  coincides  with  the  interviewers'  reports  of  having  diffi- 
culty in  the  unstructured  approach  of  remembering  all  the  items  of  infor- 
mation to  be  covered. 

Study  P2     - 

The  second  of  the  panel  validation  operations,  this  study  was  designed 
primarily  to  validate  time  deposits,  as  will  be  discussed  in  Chapter  IV.  At 
the  same  time,  it  proved  feasible  to  use  the  same  procedure  followed  in 
Study  PI  to  attempt  secondary  validation  of  nonmortgage  debt.  As  with 
Study  PI,  five  waves  of  interviews  were  scheduled  with  the  panel  members 
in  this  operation,  the  interval  between  interviews  being  roughly  three  to 
four  months.  Once  more,  certain  controlled  experiments  were  incorpo- 
rated into  the  study,  four  on  the  first  wave,  as  follows: 
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( 1 )  Use  of  a  straightforward  appeal  in  the  advance  letter  based  on  the 
need  for  information  on  how  people  handle  their  finances  versus  the  same 
letter  supplemented  with  an  attempt  to  involve  the  panel  member  further 
by  seeking  his  suggestions  and  aid  on  the  conduct  of  the  interview.  (Copies 
of  the  advance  letters  are  shown  in  Appendix  A.) 

(2)  Use  of  questionnaire  forms  in  the  interview  versus  use  of  only  re- 
minder cards  pasted  into  the  interviewer's  notebook.  This  was  a  follow-up 
of  the  third  experiment  of  the  first  wave  on  Study  PI  (p.  77),  designed  to 
cope  with  interviewer  difficulties  in  remembering  all  of  the  details  to  be 
obtained  for  each  holding. 

(3)  Request  for  dollar  holdings  as  of  the  date  of  the  interview  versus 
request  only  for  changes  that  had  taken  place  in  each  holding  during  the 
preceding  three  months. 

(4)  Request  for  information  on  all  assets  and  debts  versus  request  for 
information  on  only  specific  assets  and  debts.  The  purpose  of  this  experi- 
ment was  to  test  the  possibility  that  more  complete  information  would  be 
obtained  if  the  interviewer  did  not  have  to  cover  the  entire  range  of  assets 
and  debts. 

As  a  result  of  these  experiments,  four  different  questionnaire  forms 
were  used  on  the  first  wave  of  interviews.  With  the  exception  of  modifi- 
cations necessitated  by  the  fourth  of  these  experiments,  the  questionnaire 
covered  the  entire  spectrum  of  consumer  assets  and  debts.  Contrary  to 
the  procedure  in  Study  PI,  sample  members  were  asked  in  all  instances 
for  dollar  figures  —  in  roughly  half  the  interviews  for  dollar  holdings  and 
in  the  remainder  for  dollar  changes.  As  on  other  panel  studies,  an  attempt 
was  made  to  avoid  confounding  of  experiments;  this  was  done  by  a  sys- 
tematic subdivision  of  the  sample  into  random  segments,  in  this  case  16, 
with  the  members  of  each  segment  receiving  a  different  combination  of 
experimental  approaches. 

The  initial  sample  size  for  this  operation  was  set  at  300  completed 
interviews.  Actually,  316  interviews  were  obtained  from  a  total  eligible 
sample  of  411.  Nonrespondents  numbered  95,  of  which  82  were  refusals 
and  13  were  noncon tacts.  Overall,  the  field  performance  appears  to  have 
been  slightly  better  than  in  the  previous  panel  operation  (Study  PI), 
with  a  contact  rate  of  97  percent,  a  response  rate  of  79  percent,  and  an 
overall  interview  rate  of  77  percent. 

Table  18  shows  the  results  of  the  debt  validation  for  the  holdings  form. 
As  were  the  previous  tables  relating  to  Study  PI,  this  table  was  prepared 
on  an  individual  debt  basis  rather  than  on  a  savings  unit  basis.  The  debts 
recorded  in  this  table  correspond  to  information  sought  for  441  sample 
members,  of  whom  320  were  interviewed,  82  refused,  and  39  were  not 
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contacted.  Debts  were  reported  for  315  interviews  and  93  nonrespondents, 

as  follows : 

Holdings  form         Change  form 

Debts  reported  by  both  R  and  /, 

although  not  necessarily  the  same 

debt(s)  in  both  cases 20  12 

Debts  reported  by  R  only 12  15 

Debts  reported  by  /  only70 45  35 

No  debts  reported  from  either  source71  1 24  145 

The  overall  results  shown  in  Table  18  are  somewhat  mixed;  neither 
source  shows  up  too  well.  Less  than  one-fourth  of  the  known  debts  of  the 
respondents,  representing  about  15  percent  of  the  aggregate  dollar  amount 
owed,  was  reported  by  both  sources.  Slightly  more  than  half  of  the  num- 
ber of  debts,  although  constituting  only  one-fifth  of  the  aggregate  dollar 
amount,  was  reported  only  by  /;  whereas  about  one-third  of  the  number, 
representing  almost  two-thirds  of  the  total  dollar  amount,  was  obtained 
only  from  the  R  reports. 

Hence,  of  the  total  known  respondent  debts,  three-fourths  of  the  num- 
ber, although  only  one-third  of  the  dollar  total,  was  accounted  for  by  I; 
and  a  little  over  half  the  number,  but  almost  four-fifths  of  the  dollar 
amount,  was  supplied  by  R.72  To  the  extent  that  additional  debts  were 
not  reported  by  either  source,  all  these  percentages  would  be  too  high. 

Columns  7  and  8  of  the  table  highlight  the  tendency  for  the  average 
debt  reported  by  R  to  be  more  than  twice  the  size  of  the  average  debt 
reported  by  /.  Consequently,  when  the  reports  from  the  two  sources  are 
combined  to  form  a  "best  estimate"  of  the  average-sized  debt,  the  I  re- 
ports provide  a  substantial  underestimate  and  the  R  reports  give  a  sub- 
stantial overestimate,  with  the  latter  coming  a  little  closer  to  the  "best 
estimate." 

The  debts  of  the  nonrespondents  appear  to  average  higher  than  the 
debts  of  the  respondents,  based  on  comparison  of  the  corresponding  I  re- 
ports (line  6  versus  lines  9  and  10).  However,  the  margin  of  error  in 
these  estimates  is  undoubtedly  much  higher  than  for  those  of  the  respon- 
dents, partly  because  of  the  smaller  sample  sizes  and  partly  because  the 

70  Includes  18  nonrespondents  on  the  holdings  form  and  8  nonrespondents  on 
the  change  form. 

71  Includes  37  nonrespondents  on  the  holdings  form  and  30  nonrespondents  on 
the  change  form. 

72  These  figures  assume  that  the  J  report  was  the  more  accurate  when  data  were 
available  from  both  sources.  This  was  an  arbitrary  assumption  necessitated  by  the 
absence  of  any  firm  basis  for  making  a  decision.  The  /  reports,  coming  from  the 
creditor,  should  have  been  more  complete  but  they  also  tended  to  be  out-of-date. 
The  R  reports,  although  frequently  based  on  records,  nevertheless  were  not  always 
fully  complete. 
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Table  19.  Alternative  Estimates  of  Percentage  Distribution 
of  Individual  Debts,  by  Size,  Study  P2 


Size  of  debt 

Reported  by 
R  and  / 

All/ 
reports 

(4) 

Alii? 

reports 

(5) 

All  reports 
combined 

(6) 

(1) 

R 

(2) 

/ 

(3) 

Under  $100 

38 

43 

14 

5 

43 
33 
24 

66 

24 

8 

3 

28 
37 
24 
11 

51 

$100  to  $499.  . . . 

27 

$500  to  $999 

15 

$1,000  and  over 

7 

Total 

100 

100 

101 

100 

100 

Base  (number  of  debts) 

21 

21 

76 

62 

117 

lack  of  interview  data  made  identification  of  all  adult  members  of  the 
savings  unit  more  difficult  and  the  overlooking  of  debt  more  likely. 

Based  on  the  21  debts  for  which  a  comparison  can  be  made  between 
the  two  sources,  little  overall  tendency  is  apparent  for  respondents  to 
understate  or  overstate  those  debts  that  are  reported.  The  average  of  these 
debts  as  reported  by  R  differs  by  only  3  percent  from  the  average  for  the 
same  debts  based  on  /.  Nevertheless,  for  more  than  one-third  of  these 
debts,  the  two  reports  differed  from  each  other  by  at  least  25  percent,  as  is 
shown  by  the  following  tabulation : 

Average  debt  as 
Number  reported  by 

R  report  relative  to  I  report  of  debts  R  I 

Within  25  percent 13  $274  $271 

Higher  by  25  percent  or  more 5  339  192 

Lower  by  25  percent  or  more 3  89  291 

The  offsetting  of  large  positive  discrepancies  by  large  negative  discrep- 
ancies is  largely  responsible  for  the  close  correspondence  between  the 
aggregates.  The  same  tendencies  operate  to  bring  the  distributions  into 
reasonably  Close  alignment  (considering  the  small  sample  sizes)  for  debt 
information  available  from  both  sources,  as  shown  in  Columns  2  and  3  of 
Table  19.  Because  of  the  more  frequent  oversight  of  large  debts  by  /  but 
the  more  frequent  omission  of  many  small  debts  by  R,  the  /  data  would 
seem  to  supply  a  somewhat  closer  estimate  of  the  distribution  of  debts  by 
size,  whereas  the  R  reports  would  seem  to  be  a  considerably  better  source 
for  the  distribution  of  cumulated  debt  by  size  bracket.  In  the  latter  case, 
the  /  reports  would  be  omitting  more  than  three-fourths  of  the  known 
debt  of  $1,000  or  more. 
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Perhaps  the  principal  reason  for  the  mediocre  performance  of  both  R 
and  /  in  covering  debts  is  evident  from  Table  20.  The  layout  of  this  table 
is  similar  to  that  of  Table  18  except  that  separate  tabulations  are  now 
provided  for  retail  debt  and  for  nonretail  debt,  the  latter  comprising  in 
this  case  exclusively  debt  to  banks,  other  savings  institutions,  and  finance 
companies.  The  pertinence  of  this  distinction  is  brought  out  by  the  follow- 
ing inferences  drawn  from  this  table : 

(1)  Retail  debts  are  generally  small  and  frequent,  whereas  nonretail 
debts  (in  this  case  exclusively  debts  with  banks  and  finance  companies) 
are  much  fewer  but  also  more  sizable.  Particularly  striking  in  this  con- 
nection is  the  fact  that  the  total  dollar  amount  of  the  39  nonretail  debts 
listed  in  the  table  aggregate  is  more  than  triple  that  of  the  102  retail  debts 
listed. 

( 2 )  The  R  reports  tend  to  be  less  comprehensive  in  the  retail  area  than 
in  the  nonretail  area,  whereas  the  reverse  is  true  for  the  I  reports. 

(3)  For  both  types  of  debt,  substantial  differences  are  apparent  in  the 
average  dollar  amounts  obtained  from  each  source.  For  both  retail  and 
nonretail  debt,  the  average  of  the  R  reports  is  considerably  above  the 
average  of  the  I  reports  —  more  so  for  retail  debt  than  for  nonretail  debt. 

(4)  The  foregoing  results  serve  to  highlight  the  general  tendency, 
which  exists  apparently  in  both  types  of  debt,  for  the  average  debt  re- 
ported by  R  to  be  considerably  higher  than  the  average  debt  reported 
by  /.  Lines  4  and  5  in  both  parts  of  the  table  bring  out  the  fact  that  debts 
picked  up  by  /  only,  although  much  more  numerous  than  the  debts  re- 
ported by  R  only  (at  least  in  the  case  of  retail  debt),  are  generally  very 
small  ones.  Thus,  the  average  retail  debt  picked  up  by  I  only  was  less 
than  one-fourth  the  size  of  the  average  retail  debt  reported  by  R  only. 
Indications  are  that  those  debts  reported  by  /  only  are  relatively  unim- 
portant and  may  have  been  omitted,  for  the  most  part,  accidentally.73 

(5)  For  debts  available  from  both  sources,  lack  of  correspondence  is 
evident  between  the  two  aggregates.  The  R  report  is  considerably  above 
the  /  report  for  nonretail  debt  and  below  the  I  report  for  retail  debt.  ( In 
the  former  case,  however,  the  sample  size  is  very  low,  with  virtually 
all  the  discrepancy  being  accounted  for  by  one  debt. )  Reports  on  roughly 
half  of  the  retail  debts  deviated  by  more  than  25  percent  from  each  other. 
Of  these  large  discrepancies,  the  R  report  is  higher  in  roughly  half  of  the 

73  There  is  the  further  possibility  that  many  of  these  debts  were  not  really 
debts  of  more  than  30  days  and  therefore  should  not  have  been  included.  Respon- 
dents were  asked  not  to  include  debts  that  were  being  paid  off  within  30  days  or 
less,  whereas  the  I  reports  included  all  debts  of  any  length.  The  nature  of  the  / 
reports  made  it  difficult  to  determine  which  debts  were  indeed  of  more  than  30 
days;  in  cases  of  doubt,  the  debts  were  included. 
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cases  and  lower  in  the  other  half.   Since  the  latter  discrepancies  are  much 
larger,  the  I  reports  yield  higher  debt  estimates : 

Average  debt  as 
Number  reported  by 

R  report  relative  to  I  report  of  debts  R  I 

Within  25  percent 7  $  72  $  70 

Higher  by  25  percent  or  more 4  123  79 

Lower  by  25  percent  or  more 3  89  291 

In  contrast  to  previous  comparisons,  it  does  not  necessarily  follow  that 
the  higher  figures  are  the  more  accurate  ones. 

The  result  of  the  lack  of  correspondence  between  the  debt  figures 
obtained  from  the  two  sources  is  evidenced  not  only  in  the  differences  in 
average  debt,  but  also  in  differing  distributions  of  debt  by  size.  The  failure 
of  R  to  report  many  small  debts  combined  with  the  failure  of  /  to  pick  up 
many  large  debts  is  reflected,  for  both  types  of  debt,  in  more  elongated 
distributions  for  R  than  for  I.  The  differences  are  particularly  striking  in 
the  case  of  nonretail  debt,  where  41  percent  of  the  debt  reports  obtained 
from  I  are  above  $500  compared  with  73  percent  of  the  debts  reported  by 
R.  To  a  lesser  extent  the  same  phenomenon  is  present  in  the  retail  debt 
distributions. 

Analysis  of  the  frequency  of  debt  report  by  the  four  experimental 
variables  (p.  81)  indicates  that  neither  use  nor  nonuse  of  questionnaire 
forms  in  the  interview  nor  covering  all  assets  and  debts  or  only  some 
assets  and  debts  made  any  perceptible  difference.  However,  considerably 
higher  frequencies  of  debt  reports  were  obtained  when  the  holdings  form 
was  used  rather  than  the  change  form  and  when  the  suggestion-and- 
evaluation  advance  letter  was  used  instead  of  a  straightforward  advance 
letter.  Thus,  21  percent  of  the  savings  units  interviewed  with  a  holdings 
form  reported  one  or  more  debts  as  compared  with  16  percent  of  those 
interviewed  with  a  change  form ;  and  2 1  percent  of  those  interviewed  after 
receiving  an  evaluation  letter  reported  debt  compared  with  17  percent 
of  the  respondents  receiving  a  regular  advance  letter.  Furthermore,  53 
percent  of  all  known  debts  —  that  is,  debts  reported  by  both  R  and  I  — 
of  those  interviewed  with  a  holdings  form  were  reported  in  the  interview 
as  compared  with  only  45  percent  of  known  debts  of  those  interviewed 
with  a  change  form.  The  margin  of  superiority  on  this  basis  of  the  evalua- 
tion letter  over  the  regular  letter  is  even  larger  —  55  percent  over  45  per- 
cent, respectively. 

Only  this  last  difference  is  statistically  significant  (at  the  .05  signifi- 
cance level).  Yet,  the  differences  are  consistent  enough  to  leave  little 
doubt  regarding  their  existence.  Even  more  significant  is  the  fact  that  the 
combination  of  the  two  approaches  yields  better  results  than  either  ap- 
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proach  alone  —  differences  that  are  consistent  on  either  basis  of  analysis, 
savings  units,  or  debts.  In  either  case,  the  most  frequent  debt  reports  were 
obtained  when  the  evaluation  letter  was  used  with  a  holdings  form,  and 
the  least  frequent  debt  reports  when  a  regular  letter  was  used  with  a 
change  form. 

Unfortunately,  the  superiority  of  the  holdings  form  and  of  the  evalua- 
tion letter  is  obtained  only  at  the  cost  of  apparent  reduced  response.  After 
an  advance  regular  letter  was  sent,  completed  interviews  were  obtained 
with  92  percent  of  those  contacted ;  the  response  rate  when  the  evaluation 
letter  was  used  amounted  to  only  68  percent.  Similarly,  completed  inter- 
views were  obtained  with  83  percent  of  those  who  were  contacted  and  who 
were  to  be  interviewed  with  a  change  form,  whereas  the  corresponding 
response  rate  with  the  holdings  form  was  72  percent.  Both  differences 
are  statistically  significant. 

Apparently,  therefore,  the  evaluation  letter  and  the  more  comprehen- 
sive holdings  form  act  to  discourage  people  who  might  otherwise  cooperate 
in  granting  an  interview.  The  people  thereby  discouraged  would  seem  to 
be  primarily  less  reliable  respondents,  with  the  result  that  the  information 
supplied  by  those  who  do  cooperate  is,  on  the  whole,  likely  to  be  more 
comprehensive. 

Comparative  Evaluation 

The  eight  debt-validation  studies  reviewed  in  this  chapter  differ  in 
numerous  respects,  relating  not  only  to  the  types  of  debt  being  validated 
but  also  to  the  circumstances  under  which  the  survey  was  conducted,  the 
conditions  of  the  interview,  and  the  types  of  data  that  were  sought.  As  a 
result,  despite  the  limited  scope  of  each,  taken  together  these  surveys  pro- 
vide much  information  on  the  accuracy  with  which  debt  is  reported  in 
consumer  surveys. 

Some  Generalizations 

Perhaps  the  most  significant  generalization  to  be  drawn  from  these 
studies  is  how  the  reliability  of  debt  information  varies  with  many  differ- 
ent characteristics,  among  which  are  the  types  of  debt  being  validated,  the 
format  of  the  questionnaire,  the  types  of  data  requested,  the  conditions  of 
the  interview,  the  abilities  of  the  interviewer,  the  personality  of  the  re- 
spondent, and  the  attitude  of  the  respondent  toward  the  study. 

At  the  same  time,  certain  general  characteristics  of  errors  in  the  report 
of  debt  appear  to  underlie  all  of  these  studies.  One  is  the  tendency  for  the 
error  in  the  sample-derived  estimate  of  average,  or  of  aggregate,  debt  to 
be  in  the  direction  of  underestimation.  This  is  true  even  in  the  case  of  the 
Federal  Reserve  Survey  of  car-buyer  debt  (Study  Fl)  in  which  the  aggre- 
gate error  was  at  times  less  than  1  percent.    It  is  also  true  of  the  study  of 
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cash  loans,  one  of  the  most  sensitive  debts,  in  which  the  error  was  far 
more  substantial.  Indeed,  the  variability  in  degree  of  underestimation 
among  the  different  studies  tends  to  support  the  view  that  the  more  sensi- 
tive is  the  particular  debt  being  covered,  the  larger  is  the  degree  of 
underestimation. 

A  second  general  phenomenon,  which  serves  largely  to  explain  the  one 
just  postulated,  is  that  the  principal  source  of  error  in  estimates  of  means 
or  of  totals  is  nonreporting  of  debt  by  respondents.  This  factor  accounted 
for  the  largest  share  of  the  error  of  underestimation  in  every  case  for  which 
allocation  of  error  by  cause  was  feasible.  The  relative  importance  of  this 
factor  increased  in  magnitude  with  increases  in  the  extent  of  underestima- 
tion, leaving  little  doubt  that  this  factor  was  primarily  responsible  for 
instances  of  substantial  underestimation  of  debt. 

A  third  general  phenomenon  is  the  tendency  for  response  errors  in  debt 
that  was  reported  to  be  relatively  small.  As  a  general  rule,  these  response 
errors  tended  toward  overstatement  rather  than  understatement,  with  the 
result  that  they  served  to  offset  partly  the  substantial  underestimation  due 
to  respondent  nonreporting  (especially  in  Studies  S2b  and  P3) . 

Still  another  general  phenomenon  is  the  tendency  for  errors  in  esti- 
mates of  distributions  of  debt  to  vary  with  the  type  of  debt.  When  com- 
parisons were  possible,  primarily  in  the  car-buyer  study  (Fl)  and  in  the 
farm  study  (P3),  it  was  found  that  the  debt  distribution  obtained  from 
the  respondent  reports  approximated  fairly  closely  the  debt  distribution 
of  the  entire  sample.  Certain  errors  of  a  consistent  nature  were  apparent, 
deriving  partly  from  the  nature  of  the  response  error  patterns,  but  the 
errors  were  not  substantial. 

These  consistent  errors  were  essentially  of  two  types  —  a  tendency  to 
underestimate  the  frequency  of  small  debts  and  a  tendency  to  overestimate 
the  frequency  of  large  debts.  Study  P2,  covering  all  consumer  nonmort- 
gage  debt  (but  only  by  ex  post  validation)  suggests  that  the  same  phenom- 
enon may  be  characteristic  of  all  debt,  particularly  because  of  the  fre- 
quent, apparently  accidental,  oversight  of  small  debts. 

A  more  tenuous  generalization  is  that  those  who  refused  or  could  not 
be  contacted  were  more  in  debt  than  those  who  were  interviewed.  Sup- 
port for  this  phenomenon  is  obtained  from  three  of  the  urban  studies 
(Fl,  PI,  and  P2)  but  is  not  supported  by  the  study  of  farm  debt  (P3). 

A  final  broad  inference  from  these  results  is  that  the  effect  of  bias  on 
the  reliability  of  estimates  may  at  times  be  more  substantial  than  its  effect 
on  the  estimates  themselves.  Indeed,  a  relatively  small  bias  in  particular 
estimates  may  be  deceptive,  serving  to  hide  a  substantial  bias  in  confidence 
intervals  obtained  with  the  sample  data.  Such  was  the  case  with  the 
Federal  Reserve  car-buyer  study  in  which,  despite  very  small  biases  in 
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estimates  of  the  population  averages,  the  usual  two-sigma  confidence  inter- 
vals based  on  the  sample  data  had  virtually  zero  probability  of  including 
the  true  population  averages,  for  three  of  the  four  characteristics  tested. 
As  a  general  working  proposition,  it  appears  that  if  a  substantial  bias 
is  present  in  sample  estimates,  confidence  intervals  based  on  the  sample 
data  are  very  likely  to  underestimate  substantially  the  true  interval.  How- 
ever, relatively  low  biases  in  sample  estimates  do  not  necessarily  indicate 
correspondingly  low  biases  in  estimates  of  the  standard  errors  of  these  esti- 
mates. Whether  or  not  the  latter  is  true  depends  on  the  variance  in  the 
population  and,  to  a  large  extent,  on  sample  size.  In  this  respect,  the 
effect  of  the  latter  tends  to  run  counter  to  the  usual  precepts  of  sampling 
theory:  the  larger  is  the  sample  size,  the  more  influential  are  the  non- 
sampling  errors  and  the  more  likely  are  the  confidence  intervals  of  the 
estimates  to  be  understated  substantially. 

Distinctive  Factors 

The  studies  provide  considerable  support  for  the  view  that  the  ac- 
curacy of  debt  reported  is  affected  by  a  host  of  different  factors  relating  in 
part  to  the  conditions  of  the  interview,  in  part  to  the  respondent,  and  in 
part  to  the  interviewer  himself.  One  of  the  most  impressive  findings  is 
that  the  type  of  approach  and  the  type  of  questionnaire  used  may  affect 
strongly  the  reliability  of  the  data.  Thus,  strong  evidence  is  provided  for 
the  superiority  of  a  sealed-envelope  technique  (Study  S4),  for  the  greater 
reliability  of  holdings  data  compared  with  data  on  change  in  debt,  and 
for  the  desirability  of  using  an  approach  whereby  the  respondent  partici- 
pates in  the  evaluation  of  the  interview  instead  of  playing  a  purely  passive 
role  (Studies  P2  and  P3).  It  may  seem  somewhat  paradoxical  that  the 
latter  two  benefits  are  obtained  only  with  the  aid  of  appreciably  lower 
response.  However,  it  is  this  lower  response  that  brings  about  the  higher 
reliability,  because  the  stricter  demands  of  the  requests  for  holdings  and 
for  evaluation  of  the  interview  serve  to  weed  out  respondents  who  ap- 
parently would  not  have  been  inclined  to  provide  accurate  information 
anyway. 

There  is  also  evidence  to  support  the  view  that  a  relatively  short  ques- 
tionnaire focusing  more  directly  on  the  particular  debt  under  study  is 
likely  to  produce  more  accurate  data,  as  is  also  true  of  giving  the  inter- 
viewer advance  knowledge  of  the  holdings  of  the  respondent.  However, 
neither  of  these  conditions  may  be  feasible  in  an  actual  study. 

The  foregoing  findings  should  not  be  taken  as  suggestive  that  all  vari- 
ations in  interview  approach  may  influence  the  accuracy  of  the  data 
supplied.  Thus,  in  the  present  studies,  no  noticeable  differences  in  ac- 
curacy were  obtained  whether  the  interview  covered  all  assets  and  debts 
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or  only  selected  assets  and  debts.  Similarly,  no  differences  were  obtained 
in  the  frequency  of  debts  reported  whether  dollar  figures  were  or  were  not 
requested. 

Accuracy  of  information  is  also  affected  by  the  characteristics  of  the 
respondent.  Although  insufficient  data  were  collected  to  permit  any  con- 
clusive inferences,  the  personality  of  the  respondent  and  his  attitude  to- 
ward the  study  may  be  more  relevant  in  this  connection  than  his  socio- 
economic characteristics. 

Study  P3  suggests  that  the  interviewer  may  not  only  influence  the 
accuracy  of  report  but  under  certain  conditions  may  be  able  to  gauge  this 
accuracy.  Interviewers  who  are  more  dominant  appear  more  likely  to 
obtain  accurate  and  complete  information,  although  partly  at  the  expense 
of  a  lower  rate  of  response.  This  is  another  manifestation  of  the  "weeding 
out"  phenomenon  noted  in  the  comparison  of  the  holdings  form  versus  the 
change  form. 

The  Validation  Process 

The  studies  summarized  in  this  chapter  serve  to  illustrate  the  difficul- 
ties encountered  in  validating  survey  data.  The  results  suggest  that  in 
certain  situations  the  institutional  reports,  although  covering  presumably 
the  very  same  holdings,  may  be  more  incomplete  than  the  consumer  inter- 
view data.  Indeed,  in  these  circumstances  the  primary  use  of  data  from 
the  external  source  would  be  not  for  validation  per  se  but  for  supplement- 
ing and  adding  to  the  data  supplied  in  the  interview.  So  far  as  validation 
in  the  case  of  debt  is  concerned,  a  central  institutional  source  —  even  one 
that  presumably  has  the  complete  support  of  the  retailers  and  lending 
institutions  in  the  area  —  may  not  provide  the  comprehensive  coverage 
needed  for  validation. 


IV.  VALIDATION  STUDIES:  TIME  DEPOSITS 


Time  deposits  and  debts  are  in  many  ways  at  the  opposite  ends  of  the 
spectrum.  Debts  are  widely  discussed,  and  their  ownership  is  not  generally 
considered  to  be  a  confidential  matter.  Ownership  of  time  deposits  is  not 
considered  confidential  either,  but  the  full  extent  and  amount  of  such  own- 
ership is  to  many  people  probably  the  most  confidential  aspect  of  their 
financial  (and  in  many  cases,  personal)  life. 

For  this  reason,  a  proper  validation  study  of  time  deposits  is,  on  the 
one  hand,  more  difficult  to  conduct  but,  on  the  other  hand,  it  may  yield 
far  more  useful  information  than  validating  almost  any  other  type  of 
financial  holding.  Indeed,  such  a  study  subjects  survey  techniques  to  the 
acid  test.  If  any  data  are  likely  to  be  in  error,  time  deposits  are  the  first 
possibility.  Hence,  to  the  extent  that  errors  in  the  reporting  of  time  de- 
posits can  be  pinpointed  and  remedial  measures  taken,  data  of  greatly 
improved  reliability  could  be  expected  for  all  types  of  financial  holdings. 

Despite  the  difficulty  in  conducting  such  studies,  several  have  been 
carried  out  in  the  past  few  years  —  most  of  them  in  this  country  and  one 
in  the  Netherlands.  We  shall  first  review  the  studies  conducted  in  the 
United  States  and  later  the  study  carried  out  in  the  Netherlands.  In  the 
last  part  of  the  chapter  an  attempt  will  be  made  to  bring  together  and 
compare  the  results  of  these  studies. 

The  Initial  Study  (Study  SI ) 74 

As  a  partial  check  on  the  Survey  of  Consumer  Finances,  the  Survey 
Research  Center  undertook  in  the  summer  of  1954  a  pilot  study  to  mea- 
sure the  reliability  of  individual  reports  of  time  deposits  and  of  changes  in 
these  accounts.  A  sample  of  accounts  was  selected  from  the  files  of  a 
savings  institution,  the  sample  being  stratified  by  size  of  account,  by  ac- 
count activity,  and  by  local  area ;  the  purpose  of  the  last-mentioned  was  to 

74  This  section  is  based  on  an  unpublished  memorandum  by  James  Morgan, 
who  directed  the  original  study.  This  and  the  other  Survey  Research  Center 
studies  described  in  this  chapter  are  discussed  in  considerably  more  detail  in 
Lansing,  Ginsburg,  and  Braaten,  op.  cit. 
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Table  21.    Percentage  Distribution  of  Number  and  Aggregate 

Amounts  in  Validated  Accounts,  by  Outcome 

of  Interview  Attempt 


Outcome  of  interview  attempt 

Percentage  of 

Amounts 

Accounts 

Nonsample  addresses 

Trusts  for  minors 

Estates  of  deceased 

1.7 
3.6 
3.5 
2.8 
3.1 
1.7 

10.0 
.5 

Armed  services  address 

Address  out  of  sample  area 

P.  O.  box  or  inadequate  address 

Other 

.5 
4.6 
1.1 
2.7 

Subtotal 

16.4 

19.4 

Sample  addresses:    nonrespondents 

Refusal 

16.9 

22.8 

7.1 

4.2 

Noncontact 

Address  not  used 

13.9 
2.5 

Subtotal 

46.8 

20.6 

Sample  addresses:    interviewed 

36.8 

60.0 

Total 

100.0 

100  0 

facilitate  interviewing.  After  this  stratification  had  been  carried  out  and 
ineligible  addresses  had  been  removed,  the  sample  consisted  of  133  names 
and  addresses. 

The  interviews  with  the  sample  members  were  relatively  short.  Ques- 
tions dealt  with  how  the  family  was  doing  financially,  the  occupation  and 
income  status  of  the  family,  and  its  debts  and  savings  in  various  forms. 
In  connection  with  saving  accounts,  sample  members  were  asked  about 
the  number  of  accounts,  the  type  of  account  (single  or  joint),  and,  if 
more  than  one  account,  whether  or  not  they  were  all  in  the  same  institu- 
tion. However,  no  attempt  was  made  to  ascertain  names  of  institutions  or, 
if  more  than  one  account,  the  amount  in  each  institution.  The  sample 
member  was  also  asked  for  some  judgment  on  the  accuracy  of  the  dollar 
figures  reported  (although  there  was  apparently  no  pressure  to  consult 
records),  and  the  interview  closed  with  a  few  standard  questions  on 
demographic  characteristics. 

Interviews  were  obtained  with  95,  or  71  percent,  of  the  sample  house- 
holds. Refusals  were  encountered  at  17  percent  of  the  households,  and  at 
1 1  percent  of  the  households  nobody  could  be  contacted. 

Comparison  of  the  respondents  with  the  refusals,  noncontacts,  and 
other  sample  categories,  shown  in  Table  21,  reveals  sharp  differences  in 
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Table  22.  Percentage  Distribution  of  Accounts, 
by  Reported  and  Actual  Size 


Actual 

(2) 

Reported 

Account  balances 
(1) 

Excluding 
nonreporters 

(3) 

Including 
nonreporters 

(4) 

$0 

40^2 

14.3 

9.1 

5.2 

10.4 

13.0 

5.2 

2.6 

30^5 
11.8 

13.5 

17.0 

17.0 

6.8 

1.7 

1.7 

23  4 

$1  to  $199 

$200  to  $499 

23.4 
9  1 

$500  to  $999 

10  4 

$1,000  to  $1,999 

$2,000  to  $4,999 

$5,000  to  $9,999 

$10,000  to  $24,999 

$25,000  and  over 

13.0 

13.0 

5.2 

1.3 

1.3 

Total 

100.0 

100.0 

100  1 

Base  number 

77 

59 

77 

average  holdings  among  these  categories  of  people.  Accounts  of  those 
interviewed  were  considerably  smaller  than  the  average  - —  this  group  ac- 
counted for  60  percent  of  the  number  of  accounts  in  the  total  sample 
(including  those  of  persons  at  ineligible  addresses)  but  for  only  37  percent 
of  the  aggregate  balance  in  these  accounts.  In  contrast,  accounts  of  re- 
fusals and  of  noncontacts  were  considerably  larger  than  the  average. 
This  was  even  true,  interestingly  enough,  of  accounts  of  sample  members 
at  eligible  addresses  that  were  not  used.  Inadvertently,  apparently,  a  selec- 
tion process  appears  to  have  been  at  work  in  the  classification  of  such 
addresses. 

The  account  balances  of  those  at  ineligible  addresses  were,  in  the  ag- 
gregate, close  to  the  sample  average.  However,  considerable  variation  is 
evident  by  subcategories,  with  trust  account  balances  for  minors  being 
much  lower  than  the  average,  whereas  accounts  of  estates,  of  armed  ser- 
vices personnel,  and  of  those  with  inadequate  addresses  had  much  higher 
balances  than  the  average. 

To  judge  by  these  differences,  therefore,  bias  was  already  present  in 
the  results  by  virtue  of  inability  to  contact  all  sample  members.  Table  22 
shows  that  appreciable  response  error  was  also  present  in  the  study.  A 
clear  tendency  is  evident  for  the  number  of  small  accounts  (including  zero 
accounts)  to  be  overstated  and  for  the  number  of  large  accounts  to  be 
substantially  understated.  Thus,  16  of  the  accounts  of  the  respondents 
contained  $5,000  or  more,  but  only  six  such  accounts  were  reported.  The 
discrepancy  decreases  somewhat  if  nonreporters  are  excluded  from  the 
distribution,  as  is  done  in  column  4  of  Table  22.    Nevertheless,  the  same 
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Table  23.  Percentage  Distribution  of  Accounts, 
by  Reported  and  Actual  Dollar  Amount  of  Change 


Amount  of  change 

Up  $1 ,000  or  more 

Up  $500  to  $999 

Up  $200  to  $499 

Up  $1  to  $199 

No  change 

Down  $1  to  $199 

Down  $200  to  $499 

Down  $500  to  $999 

Down  $1,000  or  more 

Total 

Base  number 


Actual 


Reported 


8.8 

2.9 

10.3 

7.4 

5.9 

5.9 

28.0 

8.8 

14.7 

39.7 

7.4 

2.9 

10.3 

14.7 

2.9 

2.9 

11.7 

14.7 

100.0 


99.9 


68 


68 


tendency  of  understatement  of  the  frequency  of  large  accounts  remains, 
not  to  mention  the  fact  that  an  actual  study  would  ordinarily  contain  no 
basis  for  identifying  nonreporters. 

Comparison  of  the  characteristics  of  those  having  a  large  response  error 
($200  or  more)  with  those  who  were  more  accurate  indicated  that  the 
following  groups  were  much  more  likely  to  report  accounts  with  large 
errors:  those  who  owned  their  own  business,  people  over  45  years  of  age, 
those  who  had  large  accounts  ($5,000  or  more),  and  those  with  active 
accounts  (one  or  more  withdrawals  in  the  previous  five  years).  To  some 
extent,  these  factors  undoubtedly  interact  with  each  other  —  for  example, 
older  people  are  more  likely  to  have  large  accounts  —  so  that  these  find- 
ings alone  do  not  necessarily  establish  the  significance  of  these  variables. 

Since  balance  data  were  requested  not  only  as  of  the  date  of  the  inter- 
view but  also  as  of  one  year  earlier,  estimates  of  change  in  the  accounts 
could  be  compared  with  the  actual  change  for  those  68  respondents  who 
reported  this  information.  The  resulting  saving  distribution,  shown  in 
Table  23,  also  appears  to  depart  substantially  from  the  actual  distribution. 
In  particular,  the  frequency  of  positive  saving  is  greatly  understated,  espe- 
cially the  frequency  of  large  positive  saving.  Thus,  10  percent  of  the 
accounts  were  reported  up  by  $500  or  more,  whereas  the  actual  percentage 
was  over  19  percent.  The  frequency  of  "no  change"  was  overstated  sub- 
stantially —  40  percent  of  the  respondents  made  this  assertion  —  although 
it  was  true  for  only  15  percent  of  the  accounts.  Inferences  with  regard 
to  aggregate  change  based  on  the  respondent's  reports  would  have  been 
highly  unreliable.  The  sample  data  suggest  that,  on  the  average,  saving 
in  these  accounts  decreased,  whereas  it  actually  increased. 
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Table  24.  Response  and  Nonresponse  Errors 
in  Reports  of  Time  Deposits,  Study  SI 


Category 

0) 


Sample  size 
(number  of 
accounts) 

(2) 


Average  size  of  account" 


Actual 

(3) 


Reported 

(4) 


Total  sample 

Respondents 

Nonreporters 

Balance  refused,  or  not  obtainable. 

Balance  reported 

Nonrespondents 

Noncontacts 

Refusals 


133 
95 
18 
17 
60 
38 
15 
23 


n.a. 
;2,922b 
2,692 

n.a. 
2,991 

n.a. 

n.a. 

n.a. 


?1,464 

l,547b 

0 

n.a. 

2,389 


a  Estimated  from  frequency  distributions  in  memorandum  by  James  Morgan. 
b  Excludes  accounts  for  which  balance  was  refused, 
n.a.    Not  available. 


A  study  of  the  characteristics  of  respondents  associated  with  inaccurate 
reports  of  change  yields  some  of  the  same  results  as  in  the  case  of  inaccu- 
rate reports  of  balances.  Inaccurate  reports  were  more  likely  to  be  given 
by  those  owning  a  business,  by  those  over  45  years  of  age,  and  by  those 
with  large  accounts.  On  the  other  hand,  active  accounts  did  not  seem  to 
be  inaccurate  more  frequently  than  inactive  accounts,  and  there  was  some 
tendency  for  persons  at  higher-income  levels  to  be  accurate  more  often 
than  others. 

The  results  of  these  various  biases  in  the  reports  of  balances  are  brought 
together  in  Table  24.  As  is  evident  from  this  table,  the  average  (or  aggre- 
gate) reported  account  understates  the  true  figure  by  about  45  percent. 
Much  of  this  is  the  result  of  nonreporting  of  accounts,  but,  in  addition, 
this  tendency  is  aggravated  by  the  understatement  of  accounts  that  were 
reported.  Unfortunately,  data  on  the  holdings  of  the  nonrespondents  are 
not  available,  so  no  allocation  of  error  is  possible  in  this  study,  other  than 
the  fact  that  90  percent  of  the  response  error  is  due  to  nonreporting. 

Reinterview  Validation  (Study  S3) 
Background 

This  was  the  first  of  several  time-deposit  validation  studies  carried  out 
for  the  Consumer  Savings  Project  by  the  Survey  Research  Center  under 
a  subcontract  arrangement.  It  was  designed  to  investigate  the  nature  of 
the  response  and  nonresponse  errors  in  time  deposits  obtained  from  con- 
sumer interviews  and  the  manner  in  which  these  errors  varied. 

In  the  latter  connection,  two  experiments  were  built  into  this  study. 
One  experiment  involved  the  use  of  a  structured  approach  with  half  of 
the  sample  and  an  unstructured  approach  with  the  other  half.    In  the 
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former  case,  interviewers  were  required  to  make  use  of  a  highly  structured 
questionnaire  (shown  in  Appendix  B  of  Lansing,  et  al.) ,  and  to  ask  indi- 
vidual questions  in  the  same  order  as  they  appeared  on  the  question- 
naire. Interviews  with  the  other  half  of  the  sample  were  conducted  by 
means  of  an  unstructured  approach,  with  the  interviewer  having  only  a 
four-page  form  on  which  to  record  dollar  figures  and  other  specified  infor- 
mation. With  this  approach,  the  interviewer  was  left  free  to  ask  whatever 
questions  seemed  most  suitable  and  to  seek  the  required  information  by 
whatever  sequence  appeared  most  appropriate. 

The  second  experiment  consisted  of  reinterviewing  the  same  respon- 
dents approximately  six  months  after  the  initial  interview.  In  this  re- 
interview,  respondents  were  asked  not  only  for  their  current  balance  but 
also  for  the  balance  as  of  the  date  of  the  first  interview.  If  there  was  any 
discrepancy  in  the  originally  reported  balance  and  the  balance  reported 
for  that  date  in  the  second  interview,  the  respondent  was  asked  to  reconcile 
the  difference.  In  this  way,  it  was  hoped  that  the  experiment  would  throw 
light  on  the  reliability  of  financial  data  obtained  for  an  earlier  period,  as 
well  as  on  the  value  of  a  reconciliation  procedure. 

For  this  study  a  probability  sample  of  account  holders  was  selected  in 
a  metropolitan  area  which,  after  removing  ineligible  names  and  addresses, 
numbered  168.  The  sample  was  restricted  to  individual  or  joint  accounts 
of  $1,000  or  more  owned  by  one  or  two  adults  living  in  the  metropolitan 
area. 

The  questionnaire  was  similar  to  that  employed  in  the  Survey  of  Con- 
sumer Finances,  the  sample  members  being  requested  to  cooperate  in  a 
study  of  money  management  and  financial  practices.  Dollar  amounts  were 
requested  for  all  assets  and  debts,  although  no  attempt  was  made  to  obtain 
names  of  institutions.  The  unstructured  interview  covered  much  the  same 
material,  except  that  the  order  of  the  questions  was  at  the  interviewer's 
discretion,  and  questions  on  other  subjects  could  be  asked  in  addition 
to  those  for  which  information  was  required. 

Interviewers  used  in  the  study  had  had  considerable  experience  with 
the  Survey  of  Consumer  Finances.  Since  they  were  accustomed  primarily 
only  to  the  structured  approach,  they  were  given  additional  training  in 
the  conduct  of  an  unstructured  interview. 

The  remainder  of  this  section  presents  the  principal  findings  obtained 
from  the  initial  interviews.  The  results  of  the  reinterviews  belong  with 
the  discussion  of  the  effects  of  a  panel  operation  and  will  therefore  ap- 
pear in  Chapter  VI. 

Response  and  Nonresponse  Errors 

The  response  rates  obtained  in  this  study  are  shown  in  column  2  of 
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Table  25.  Average  Size  of  Account,  by  Outcome 
of  Interview  Attempt,  Study  S3 


Number 

of 
accounts 

(2) 


168 
109 
28 
20 
61 
59 
34 
25 


Average  size  of 
account 


Actual 

(3) 


£3,579 
3,809 
4,212 
4,768 
3,310 
3,122 
3,155 
3,076 


Reported 

(4) 


$2,105 

2,105 

0 

2,832 


Percentage 
of  deviation, 

reported 
from  actual* 

(5) 


41.1 
44.7 
100.0 

14 .4 


Category 

(1)_ 

Total  sample 

Respondents 

Nonreporters 

Balance  refused 

Balance  given 

Nonrespondents 

Refusals 

Other 

a  Actual  —  reported 
Actual 


Table  25.  Interviews  were  carried  out  with  65  percent  of  the  168  sample 
members;  20  percent  refused  and  15  percent  either  could  not  be  contacted 
or  were  not  approachable  for  some  other  reason. 

Column  3  shows  that  the  account  balances  of  the  nonrespondents 
averaged  considerably  below  those  of  the  respondents,  which  was  con- 
trary to  the  findings  of  Study  SI.  However,  the  accounts  of  the  nonrespon- 
dents averaged  considerably  above  the  reported  average  size  of  accounts 
of  the  respondents.  As  a  result,  the  reported  accounts  of  the  respondents 
tend  to  understate  the  actual  accounts  of  the  nonrespondents,  even  though 
the  true  situation  is  the  reverse.  (In  this  respect,  it  is  interesting  to  note 
the  little  difference  between  the  average  size  of  the  accounts  of  the  re- 
fusals and  that  of  the  accounts  of  the  noncontacts. ) 

As  was  the  case  in  the  previous  study,  Table  25  indicates  the  presence 
of  substantial  response,  as  well  as  nonresponse,  errors.  Of  the  109  respon- 
dents, 26  percent  denied  having  an  account,  and  an  additional  18  percent 
reported  an  account  but  refused  to  give  the  balance.  Furthermore,  of  the 
55  percent  who  did  report  an  account  and  the  balance,  the  average  under- 
stated the  true  figure  by  14  percent. 

If  we  combine  the  understatements  of  those  who  did  report  a  balance 
with  the  failure  of  many  respondents  to  report  an  account,  the  average 
reported  size  of  account  is  seen  to  understate  the  actual  figure  by  roughly 
45  percent.75    Since  the  balances  of  the  nonrespondents  were  less  than 

75  This  estimate  is  obtained  on  the  assumption  that  the  average  size  of  account 
of  those  refusing  to  state  a  balance  is  equivalent  to  the  average  size  account  of 
those  who  did  report  a  balance.  This  is  a  somewhat  extreme  assumption  since,  at 
least  by  hindsight,  there  is  a  strong  basis  for  believing  that  those  who  refuse  dollar 
figures  generally  have  more  than  those  who  report  dollar  figures. 
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those  of  the  respondents,  the  overall  understatement  is  somewhat  less, 
namely,  about  40  percent. 

The  error  in  these  estimates  can  be  allocated  among  four  causes: 
error  due  to  faulty  estimates  of  the  balances  of  the  nonrespondents,  error 
due  to  the  nonreporters,  error  due  to  inaccurate  estimates  of  balances  re- 
fused, and  response  error.76  In  view  of  the  consistent  understatement,  it 
is  not  surprising  to  find  that  in  this  case  all  the  components  of  error  are 
positive,  and  thus  each  of  these  four  factors  serves  to  magnify  the  extent 
of  error  in  the  overall  estimate.  The  allocation  is  as  follows : 

Error  in  estimated  balances  of  the  nonrespondents 23 . 1% 

Error  due  to  the  nonreporters 48 . 8 

Error  in  account  estimates  of  those  refusing  balances 16.0 

Response  error 12.1 

100.0% 

Hence,  failure  to  report  the  existence  of  accounts  is  again  the  principal 
cause  of  error,  in  this  instance  accounting  for  roughly  half  of  the  total. 
Contrary  to  the  findings  of  previous  studies,  error  in  the  estimated  bal- 
ances of  the  nonrespondents  is  also  fairly  substantial,  being  compounded 
of  a  sizeable  understatement  of  accounts  by  the  reporters  and  a  relatively 
large  number  of  nonrespondents. 

The  substantial  understatement  in  the  average  balance  also  serves  to 
produce  a  sizeable  underestimate  of  the  true  standard  error  of  the  mean. 
The  pertinent  data  are  shown  at  the  top  of  the  following  page. 

76  The  error  allocation  is  based  on  the  following  segmentation,  similar  to  that 
used  previously.  Denote  relative  proportions  of  sample  members  in  particular  cate- 
gories by  P,  average  actual  balance  by  A,  and  average  reported  balance  by  X, 
with  subscripts  representing  the  following:  s,  nonrespondents;  o,  respondents,  ac- 
count not  reported;  l,  respondents  reporting  account  but  refusing  balance;  2, 
respondents  reporting  account  and  balance.  The  actual  average  balance  per  sample 
member  is,  then: 

A  =  PsAs  +  P0A0  +  P.A,  +  P2A2. 

The  estimated  average  balance  is: 

X  =  PsXs  +  PtXi  +  P2X2, 


where 


x'~p,  +  p7+p,x* 


on  the  assumption  that  average  balances  are  attributed  to  nonrespondents  on 
the  basis  that,  (a)  the  proportion  of  nonrespondents  owning  accounts  is  the  same 
as  the  (reported)  proportion  of  respondents  owning  accounts,  and  (b)  the  average 
balances  of  nonrespondents  account  owners  is  the  same  as  that  reported  by  P°. 

The  error  (E)  is,  then: 

E  =  Pa  {A,  —  Xs)  +  P0A0  +  Pi  {Ax  —  X2)  +  P2  (At  —  X,), 
which  serves  as  the  basis  for  the  error  allocation. 
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k  =  E/<jy 6.7 

r -.59 

Apparent  standard  error  of  the  mean  (cry) $     253 

True  standard  error  of  the  mean  (\/ MSE) $1 ,  727 

Error-free  standard  error  (<Tx) $274 

The  reason  for  the  very  high  value  of  the  true  standard  error  is  the 
dominance  of  the  expression  by  the  bias  in  the  estimate  of  the  mean, 
$1,705.  The  negative  correlation  between  the  reporting  errors  and  the 
true  values,  —.59,  is  itself  the  result  of  the  high  incidence  of  nonreporting, 
which  serves  to  assign  all  the  nonreporters  opposite  values  of  X  and  E. 
This  same  phenomenon  is  responsible  for  causing  the  standard  error  of 
the  mean  in  the  absence  of  response  error  to  exceed  the  apparent  standard 
error:  the  latter  is  spuriously  low  because  of  the  many  zero-reported 
balances. 

The  substantial  understatement  in  account  balances  also  results  in  the 
size  distribution  of  accounts  as  obtained  from  the  interview  data  greatly 
overstating  the  frequency  of  small  accounts  and  understating  the  frequency 
of  large  accounts,  as  shown  in  Table  26.  If  nonreporters  are  included, 
about  45  percent  of  the  accounts  were  reported  to  contain  less  than  $1,000, 
whereas  the  true  figure  for  this  group  was  less  than  4  percent.77  Even  if 
nonreporters  are  excluded,  about  18  percent  of  the  respondents  still  alleged 
the  balance  in  their  account  to  be  less  than  $1,000. 

At  the  other  end  of  the  scale,  about  28  percent  of  the  respondents  had 
over  $5,000  in  their  accounts,  but  accounts  of  this  size  were  reported  by 
a  little  over  12  percent  (including  those  who  refused  dollar  figures),  and 
by  18  percent  of  respondents  who  reported  a  dollar  balance. 

Table  26  also  brings  out  rather  strikingly  the  much  higher  frequency 
of  large  accounts  both  among  nonreporters  and  among  those  who  refused 
balances  than  among  those  who  gave  dollar  figures.  Thus,  whereas  18 
percent  of  the  latter  had  accounts  of  more  than  $5,000,  this  was  true  of 
36  percent  of  the  nonreporters  and  of  45  percent  of  those  who  refused 
balances. 

Further  investigation  revealed  a  number  of  relationships  between  re- 
sponse and  nonresponse  errors  and  other  variables.  In  particular,  people 
with  smaller  accounts  were  more  likely  to  report  the  account  and  to  report 
it  accurately.  A  similar  tendency,  although  less  strong,  was  apparent  for 
income  level,  accuracy  being  higher  among  those  with  lower  incomes.  In 
addition  —  not  surprisingly  in  view  of  the  preceding  results  —  greater  ac- 
curacy was  associated  with  increased  education  and  with  white-collar 
rather  than  blue-collar  occupational  status.   On  the  other  hand,  there  was 

77  People  with  accounts  of  less  than  $1,000  at  the  time  of  sample  selection 
were  excluded.  However,  the  validation  was  carried  out  as  of  the  date  of  the 
interview,  so  that  it  was  possible  for  some  accounts  to  be  actually  less  than  $1,000. 
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Table  26.    Percentage  Distribution  of  Accounts  of  Respondents, 
by  Size  and  Type  of  Response,  Study  S3 


Actual 

Reported 

Size  of  account 
(1) 

Total 

(2) 

Non- 
reporters 

(3) 

Balance 
refused 

(4) 

Balance 
reported 

(5) 

Exclud- 
ing non- 
reporters 

(6) 

Includ- 
ing non- 
reporters 

(7) 

$0 

3^7 

13.8 

18.3 

14.7 

14.7 

7.3 

11.9 

7.3 

8.3 

i'.i 

14.3 
14.3 
14.3 
14.3 
14.3 
14.3 
7.1 

10.0 

15^0 
5.0 
20.0 
5.0 
20.0 
10.0 
15.0 

3^3 

21.3 

21.3 

18.0 

13.1 

4.9 

8.2 

3.3 

6.6 

13.1 
4.9 

24.6 

11.5 
9.8 

18.0 

8^2 
9.8 

31.5 

$1  to  $499 

$500  to  $999 

$1,000  to  $1,499.  .  . 
$1,500  to  $1,999.  .  . 
$2,000  to  $2,999.  .  . 
$3,000  to  $3,999.  .  . 
$4,000  to  $4,999 .  .  . 
$5,000  to  $6,999 .  .  . 
$7,000  to  $8,999.  .  . 
$9,000  or  more 

9.0 
3.4 

16.8 
7.9 
6.7 

12.4 

5.6 
6.7 

Total 

100.0 

100.0 

100.0 

100.0 

99.9 

100.0 

Base  number 

109 

28 

20 

61 

61 

89 

Average  balance.  .  . 

$3,809 

$4,212 

$4,768 

$3,310 

$2,832 

$1 ,941 

little  evidence  that  age  of  the  respondent  or  sex  was  related  to  accuracy 
of  report. 

Respondents  who  reported  keeping  records  were  also  more  likely  to 
report  accurate  data  than  those  who  reported  not  keeping  records  (al- 
though no  pressure  was  put  on  the  respondents  to  consult  these  records 
or  a  passbook  in  giving  dollar  figures) .  People  who  used  their  accounts 
frequently  were  less  likely  to  make  errors,  particularly  in  regard  to  over- 
stating the  true  figure. 

The  structured  approach  was  more  successful  than  the  unstructured 
one.  The  response  rate  in  the  former  case  was  considerably  higher  (77 
percent  versus  54  percent)  and  response  errors  were  less  numerous.  Never- 
theless, the  significance  of  this  finding  remains  doubtful  because,  in  retro- 
spect, the  interviewers  appear  to  have  been  unsure  of  themselves  in  using 
the  unstructured  approach  and  do  not  seem  to  have  received  sufficient 
training  to  have  enabled  them  to  handle  the  many  different  problems  that 
arose  in  the  use  of  this  approach.78 

A  Third  Validation  Study  (Study  S5) 
Background 

Study  S5  was  carried  out  by  the  Survey  Research  Center  about  a  year 
78  Lansing,  Ginsburg,  and  Braaten,  op.  cit.,  pp.  61-62. 
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after  Study  S3  and  concurrently  with  the  study  of  cash  borrowers  (Study 
S4).  Its  purpose  was  to  explore  further  the  factors  contributing  to  re- 
sponse error  in  reports  of  time  deposits  and  to  see  if  certain  additional 
methods  might  not  help  reduce  this  error. 

The  sample  was  selected  from  account  holders  in  another  metropolitan 
area  and  was  limited  to  accounts  between  $500  and  $15,000  in  the  names 
of  adults.  As  in  the  study  of  cash  loans,  the  sample  included  a  number  of 
names  and  addresses  selected  from  the  area  telephone  directory,  partly  to 
safeguard  further  the  anonymity  of  the  names  and  addresses  coming  from 
the  institutional  sources  and  partly  to  make  the  survey  more  realistic  from 
an  interviewing  standpoint.  After  the  completed  questionnaires  were 
turned  in,  those  for  people  selected  from  the  telephone  directory  were  ex- 
cluded from  the  analysis. 

Two  experimental  approaches  were  tested  in  this  study.  One  approach 
involved  interviewing  the  head  of  the  household  half  the  time  and  the 
wife  of  the  head  the  rest  of  the  time.  The  second  approach  consisted  of 
offering  a  $10  payment  on  half  of  the  interviews  but  not  on  the  other 
half.  As  before,  these  experiments  were  carried  out  by  means  of  a  ran- 
domized factorial  design,  in  this  case  dividing  the  sample  into  four  equal 
random  segments  and  using  a  different  combination  of  approaches  with 
each  segment. 

The  questionnaire  was  a  general  one  relating,  as  in  Study  S3,  to  dif- 
ferent aspects  of  financial  management  and  money-handling  practices.  A 
structured  approach  was  used  in  all  interviews.  Financial  data  were  to  be 
supplied  on  forms  which  the  respondent  filled  in  and  sealed  himself  —  the 
"sealed-envelope"  technique  used  in  Study  S4. 

An  additional  feature  was  the  use  of  a  follow-up  mail  questionnaire  to 
obtain  data  on  respondent  reactions  to  the  interview.  In  this  question- 
naire, questions  were  asked  to  probe  the  attitude  of  the  respondent  to  the 
interviewer,  to  the  study  and  its  sponsor  (the  Survey  Research  Center), 
and  to  the  interview  itself.  The  questions  asked,  and  their  relationship  to 
these  categories,  were  as  follows : 79 

(1)   Forces  arising  out  of  the  interpersonal  relation  between  the  respondent 
and  the  interviewer. 

(a)  General  attitude  toward  the  interview  as  a  whole. 

Question:   How  would  you  feel  about  being  interviewed  again? 

(b)  Attitude  toward  the  interviewer. 

Question:  How  did  you  like  the  interviewer  as  a  person? 

(c)  Attitudes  toward  the  content  of  the  interview. 

Questions:  How  interesting  did  you  find  the  interview?  How  well 
did  the  interviewer  succeed  in  making  clear  to  you  what  the  study 
was  about? 

10  Ibid.,  pp.  154-55. 
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(2)  Forces  arising  out  of  the  relation  of  the  respondent  to  the  project  and 
its  sponsor  and  to  the  University. 

Questions:  How  do  you  feel  about  whether  surveys  like  this  one 
are  a  good  idea?  During  the  interview  did  you  have  any  doubts  as 
to  whether  you  should  be  giving  the  information? 

(3)  Forces  arising  out  of  other  attitudes  of  the  respondent. 

Questions:  Were  there  any  questions  which  you  thought  were  too 
personal  or  prying?  Do  you  think  people  will  give  us  accurate  in- 
formation about  their  finances  or  not? 

Estimates  of  Holdings 

After  eliminating  ineligible  names  and  addresses,  the  sample  consisted 
of  153  family  units.  Of  this  number,  interviews  were  completed  with  only 
58  percent,  due  primarily  to  refusal  to  grant  an  interview  by  one  out  of 
every  three  of  those  contacted.  By  comparison,  the  refusal  rate  was  23 
percent  in  Study  S3,  17  percent  in  Study  SI,  and  less  than  9  percent  in 
the  cash-loan  study,  Study  S4. 

Two  reasons  are  advanced  to  explain  the  higher  refusal  rate  of  Study 
S5.  One  was  the  lesser  degree  of  direct  supervision.  At  the  time  of  the 
study,  the  Survey  Research  Center  did  not  have  any  field  supervisors  in 
that  area  (contrary  to  the  situation  in  the  areas  where  preceding  studies 
had  been  conducted),  so  that  these  interviewers  were  left  more  to  their 
own  resources.  Second,  considerable  interviewer  dissatisfaction  was  re- 
ported with  the  experiment  involving  an  offer  of  cash  to  the  respondent. 
Although  this  should  not  of  itself  affect  the  likelihood  of  obtaining  an 
interview,  substantial  differences  appear  nevertheless  in  the  response  rates 
of  sample  members  who  were  offered  payment  as  compared  with  those  who 
were  not  offered  payment.  Interviewers  in  the  former  situation  managed 
to  interview  two-thirds  of  the  sample  members,  or  three-fourths  of  those 
contacted ;  whereas  among  sample  members  who  were  offered  a  cash  pay- 
ment, interviews  were  carried  out  with  only  60  percent  of  those  contacted. 
As  was  the  case  in  the  experiment  with  the  unstructured  approach,  the 
fact  that  the  interviewer  was  going  to  be  doing  something  which  was  either 
disagreeable  or  for  which  the  reason  was  not  clear  apparently  influenced 
his  approach  to  the  respondent  and,  accordingly,  the  likelihood  of  obtain- 
ing the  interview. 

From  a  validation  point  of  view,  Table  27  indicates  that  the  high  rate 
of  nonresponse  was  doubly  unfortunate  because,  as  in  previous  studies,  the 
holdings  of  the  nonrespondents  appear  to  have  differed  appreciably  from 
those  of  the  respondents.  In  this  study,  the  accounts  of  the  nonrespondents 
averaged  higher  than  those  of  the  respondents,  so  that  any  underestima- 
tion of  holdings  by  the  respondents  would  be  magnified  when  used  as  a 
basis  to  estimate  the  holdings  of  the  nonrespondents. 
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Table  27.  Average  Size  of  Account.,  by  Outcome 
of  Interview  Attempt,,  Study  S5 


Category 

(1)_ 

Total  sample 

Respondents 

Nonreporters 

Balance  refused 

Balance  given 

Nonrespondents 

Refusals 

Other 


Number 

of 
accounts 

(2) 


153 
89 
22 
13 
54 
64 
44 
20 


Average  size  of 
account 


Actual 

(3) 


52,572 
2,394 
2,168 
3,273 
2,274 
2,819 
2,704 
3,072 


Reported 

(4) 


H,742 

1,742 

0 

2,314 


Percentage 
of  deviation, 

reported 
from  actual 

(5) 


32.3 
27.2 
100.0 

-1.8 


Table  27  shows  substantial  understatement  of  holdings  by  the  respond- 
ents, primarily  because  of  the  failure  of  one-fourth  of  those  interviewed 
to  report  the  validated  account.  In  addition,  those  who  refused  to  report 
a  balance,  representing  15  percent  of  those  interviewed,  had  substantially 
higher  holdings,  on  the  average,  than  did  those  who  reported  balances. 
Hence,  even  though  response  errors  among  those  who  reported  balances 
were  negligible,  the  estimated  average  reported  account  size  understates 
the  actual  figure  by  27  percent.  Since  the  holdings  of  the  nonrespondents 
averaged  higher  than  those  of  the  respondents,  the  estimated  average  ac- 
count balance  for  the  entire  sample  understates  the  true  figure  by  even 
more,  32  percent. 

The  allocation  of  this  error  by  the  four  principal  sources  of  error  is 
as  follows : 

Error  in  estimated  balances  of  the  nonrespondents 54. 3% 

Error  due  to  nonreporters 37.6 

Error  in  account  estimates  of  those  refusing  balances 9.8 

Response  error —1.7 

100.0% 

In  contrast  to  most  of  the  previous  validation  studies,  the  principal 
source  of  error  in  this  case  is  faulty  estimates  of  the  account  balances  of 
the  nonrespondents  - —  stemming  in  part  from  the  large  proportion  of  the 
sample  in  this  category  and  in  part  from  the  substantial  difference  between 
their  average  holdings  and  those  reported  by  the  respondents.80  Failure 
to  report  accounts  is  the  other  principal  source  of  error. 

80  Note  that  this  error  depends  to  a  large  extent  on  the  difference  between  the 
actual  holdings  of  the  nonrespondents  and  the  reported  holdings  of  the  respon- 
dents. The  latter  in  turn  is  affected  substantially  by  nonreporting.  The  result  is 
that  even  if  those  who  report  balances  are  fully  accurate,  and  even  if  the  holdings 
of  the  respondents  correspond  with  those  of  the  nonrespondents,  error  in  this 
component  will  appear  to  the  extent  that  accounts  are  not  reported. 
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Table  28.  Percentage  Distribution  of  Actual  and 
Reported  Accounts,,  by  Size,,  Study  S5 


Size  of  account 


(1) 


Actual 


(2) 


Reported8 


Excluding 
nonreporters 

(3) 


Including 
nonreporters 

(4) 


SI  to  $499 

$500  to  $999 

$1,000  to  $1,999 

$2,000  to  $2,999 

$3,000  to  $4,999 

$5,000  or  more 

Total 

Base  number 

a  Excludes  refusals  of  balances  (13) 


13.5 
22.5 
20.2 
16.8 
12.4 
14.6 


100.0 


16.7 
22.2 
14.8 
16.7 
16.7 
12.9 


100.0 


29.0 
11.8 
15.8 
10.5 
11.8 
11.8 
9.2 


99.9 


89 


54 


76 


The  low  degree  of  response  error  is  reflected  in  a  somewhat  better 
comparison  of  the  reported  distribution  of  accounts  by  size  with  the  actual 
distribution,  as  is  evident  from  Table  28.  Nevertheless,  the  relatively  high 
frequency  of  nonreporting  leads  to  systematic  understatement  of  the  fre- 
quency of  large  accounts  and  to  some  overstatement  of  the  frequency  of 
small  accounts.  This  is  particularly  true  of  the  reported  distribution  as 
compared  with  the  distribution  of  all  validated  accounts  by  size  because 
of  the  greater  relative  proportion  of  large  accounts  among  the  nonrespon- 
dents. 

The  standard  error  of  the  mean  is  again  greatly  understated,  as  is 
apparent  from  the  following  data : 

k  =  E/ay 2.8 

r -.56 

Apparent  standard  error  of  the  mean  (op) $229 

True  standard  error  of  the  mean  (y/MSE) $698 

Error-free  standard  error  (o"x) $253 

In  addition,  the  standard  error  in  the  absence  of  response  error  is  again 
somewhat  higher  than  the  apparent  standard  error,  and  for  the  same 
reason  as  before  —  the  high  frequency  of  nonreporters. 

Investigation  of  the  factors  associated  with  response  error  supports 
previous  findings  that  greater  reliability  (more  frequent  reporting  of  the 
account  as  well  as  more  accurate  reporting)  was  associated  with  higher 
income,  higher  education,  white-collar  rather  than  blue-collar  occupa- 
tional status,  and  possibly  younger  age  group.  No  appreciable  differences 
in  reliability  were  found  for  sex.    A  number  of  variables  other  than  the 
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demographic  ones  were  also  associated  with  reliability  of  report.  These 
included  whether  or  not  records  of  family  expenditures  were  kept,  whether 
or  not  the  respondent  reported  dollar  figures  (for  those  who  did  so)  in 
round  numbers,  and  whether  or  not  the  family  norms  were  in  the  direction 
of  free  discussion  of  financial  subjects.81  In  addition,  a  significant  correla- 
tion was  obtained  between  interviewers'  ratings  of  respondent  accuracy 
and  both  frequency  of  report  of  the  validated  account  and  the  response 
error  when  a  dollar  figure  was  given.  On  the  other  hand,  no  relationship 
was  found  between  reliability  of  report  and  the  size  of  the  account,  the 
offer  of  a  cash  payment,  or  the  date  for  which  the  balance  was  reported. 
Virtually  the  same  frequency  of  nonreporting,  as  well  as  of  inaccurate 
reporting,  was  obtained  when  balances  were  requested  eighteen  months 
prior  to  the  interview  as  when  balances  were  requested  six  months  prior 
to  the  interview. 

Estimates  of  Change 

Since  balances  were  requested  from  the  respondents  as  of  both  Jan- 
uary 1,  1958,  and  January  1,  1959,  the  saving  of  the  respondents  in  these 
accounts  could  be  estimated  by  taking  the  difference  between  the  two 
figures.  This  proved  feasible  for  48  respondents  who  reported  balance 
figures  on  both  dates.  To  judge  by  columns  2  and  3  of  Table  29,  the 
distribution  of  these  48  respondents  by  amount  of  change  corresponded 
reasonably  well  with  that  of  all  respondents;  the  only  appreciable  dif- 
ference is  a  much  higher  proportion  of  the  latter  who  experienced  no 
change  in  the  account  balances. 

Comparison  of  the  actual  and  reported  change  distributions  for  the 
48  respondents  for  whom  such  a  comparison  could  be  made  indicates  that 
with  one  exception  the  two  distributions  do  not  differ  appreciably  from 
each  other  (columns  3  and  4  of  Table  29).  The  exception  lies  in  the 
much  higher  reported  frequency  of  no  change:  about  one-fifth  of  the 
respondents  made  such  an  allegation  although  this  was  actually  true  for 
none  of  them.82  Large  changes,  particularly  large  increases,  were  also 
somewhat  understated  in  this  study,  as  in  the  other  validation  studies.  Of 
the  nine  changes  involving  $500  or  more,  seven  were  reported  in  the  same 
interval,  but  the  other  two  were  understated  substantially. 

81  The  latter  was  ascertained  by  asking  the  respondent  how  old  he  was  before 
he  knew  the  family  income  and  how  old  he  thought  children  should  be  before 
being  told  about  family  income  and  savings.  The  replies  to  both  questions  ex- 
hibited a  strong  inverse  correlation  between  reliability  of  report  and  age. 

82  In  two  of  these  instances,  the  balance  increased  less  than  $100,  which  might 
be  regarded  as  a  negligible  difference.  In  six  other  instances,  the  balance  increased 
between  $100  and  $500;  one  balance  increased  between  $500  and  $1,000;  and  the 
remaining  balance  decreased  between  $500  and  $1,000. 
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Table  29.   Percentage  Distribution  of  Accounts,  by  Reported  and 
Actual  Amount  of  Change,,  Calendar  1958,  Study  S5 


Amount  of  change 


(1) 

Up  $2,000  or  more 

Up  $1,000  to  $1,999... 

Up  $500  to  $999 

Up  $100  to  $499 

Up  $1  to  $99 

No  change 

Down  $1  to  $99 

Down  $100  to  $499.  .  . 
Down  $500  to  $999 .  .  . 

Total 

Base  number 


Actual 


All 

respondents 

(2) 


4.6 
10.5 
10.5 
29.1 
18.6 

10.5 

3.5 
7.0 
5.8 


100.0 


Balances 
reported  on 
both  dates 

(3) 


4.2 
10.4 

8.3 
41.7 
16.7 

0 

6.2 
8.3 
4.2 


100.0 


Reported 

(4) 


6.2 

8.3 

14.6 

20.9 

12.5 

20.9 

4.2 
6.2 
6.2 


100.0 


86 


48 


48 


The  Mail  Questionnaire 

The  mail  questionnaire  described  on  page  102  was  sent  to  all  of  the 
respondents  within  two  weeks  after  the  interview.  The  response  was  very 
good,  with  80  percent  returning  the  questionnaire;  this  was  the  highest 
response  rate  achieved  to  date  in  a  number  of  such  mail  follow-up  studies 
of  the  Survey  Research  Center. 

Analysis  of  the  returns  indicated  that  on  all  eight  questions  a  favorable 
response  was  associated  with  a  higher  likelihood  of  reporting  the  account, 
as  is  shown  in  Table  30.  In  addition,  reporting  the  existence  of  an  account 
but  refusing  the  balance  was  much  more  frequent  among  those  who  were 
influenced  unfavorably  by  the  interview  than  among  other  respondents. 
Contrary  to  expectations,  however,  among  those  who  did  report  dollar 
balances,  frequency  of  accurate  reporting  was  no  higher  among  those 
favorably  influenced  by  the  interview  than  among  those  unfavorably 
influenced. 

Two  Panel  Studies 
Background 

Time  deposits  were  validated  in  two  panel  studies  of  the  Consumer 
Saving  Project.  Since  the  methods  used  in  each  study  were  similar,  the 
results  will  be  presented  jointly. 

Time  deposits  served  as  a  primary  validation  source  for  the  urban 
panel  study  (P2),  all  the  sample  members  having  been  selected  by  random 
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Table  30.  Distribution  of  Mail  Respondents,  by  Reporting 
of  Account  and  Answers  to  Mail  Questionnaires 


Question 

Response81 

Percent- 
age of 
total 

Number 

of 
nonre- 
porters 

Number 
refus- 
ing 
balance 

Number 
report- 
ing 
dollar 
figure 

Clarity  of  interviewer 

explanation 

Favorable  (60) 

100 

25 

10 

65 

Unfavorable  (11) 

100 

27 

18 

55 

Interest  in  interview.  .  .  . 

Favorable  (49) 

99 

26 

6 

67 

Unfavorable  (22) 

101 

23 

23 

55 

Attitude  toward  possible 

reinterview 

Favorable  (55) 

101 

26 

7 

68 

Unfavorable  (16) 

100 

25 

25 

50 

Attraction  of  interviewer 

as  a  person 

Favorable  (59) 

100 

27 

10 

63 

Unfavorable  (12) 

99 

16 

17 

56 

Value  of  such  surveys .  .  . 

Favorable  (63) 

99 

25 

8 

66 

Unfavorable  (8) 

101 

25 

38 

38 

Were  questions  too 

personal? 

Favorable  (43) 
Unfavorable  (28) 

100 

28 

5 

67 

99 

21 

21 

57 

Doubts  about  giving 

information 

Favorable  (30) 

100 

27 

3 

70 

Unfavorable  (41) 

100 

25 

17 

58 

Likelihood  of  other 

people  giving  ac- 

curate financial 

information 

Favorable  (34) 

101 

24 

9 

68 

Unfavorable  (37) 

101 

27 

14 

60 

a  Figures  in  parentheses  are  number  of  people  giving  particular  response. 

Source:  J.  B.  Lansing,  G.  P.  Ginsburg,  and  Kaisa  Braaten,  An  Investigation  of  Response 
Error,  Studies  in  Consumer  Savings,  No.  2  (Urbana:  University  of  Illinois  Bureau  of  Economic 
and  Business  Research,  1961),  pp.  159-61. 


probability  methods  from  the  files  of  savings  institutions.  Since  the  design 
of  this  study  was  described  in  the  preceding  chapter,  it  is  not  necessary 
to  repeat  the  details.  However,  it  is  pertinent  to  note  that  the  question- 
naires used  were  general  in  scope,  covering  many  assets  and  debts  in 
addition  to  time  deposits.  Also,  in  contrast  to  the  procedure  in  previous 
studies  covered  in  this  chapter,  questions  were  asked  about  names  of  insti- 
tutions in  all  instances,  so  that  the  problem  of  account  identification, 
although  not  eliminated,  was  simplified  considerably. 

Stratification  of  account  size  was  used  in  selecting  the  sample  members. 
At  the  time  of  sample  selection,  equal  numbers  of  sample  members  had 
accounts  in  each  of  the  following  categories:  under  $100;  $100  to  $1,000; 
$1,000  to  $5,000;  and  $5,000  or  more.  As  was  the  case  in  the  previous 
studies,  accounts  were  limited  to  those  held  by  individuals  only. 

Because  of  the  split-run  experiment  involving  asking  half  of  the  panel 
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members  for  holdings  and  half  for  change,  account  balances  were  not 
obtained  for  half  of  the  sample  on  the  first  wave.  Hence,  the  analysis  that 
follows  deals  separately  with  those  who  were  asked  about  holdings  and 
those  who  were  asked  about  change,  as  was  done  in  the  case  of  debt. 
Furthermore,  since  the  focus  of  this  chapter  is  on  single-interview  studies, 
the  emphasis  is  on  the  reports  of  balances,  the  only  reports  which  could  be 
validated  on  the  first  wave. 

The  other  panel  validation  study  (P3)  was  conducted  in  a  farm  area. 
Savings  accounts  served  as  a  primary  validation  source  for  74  of  the  409 
eligible  sample  members  (farmers)  in  the  study.  As  was  noted  in  the 
initial  description  of  this  study  (pages  64-75),  some  of  these  respondents 
were  to  be  interviewed  with  a  holdings  form  and  others  with  a  change 
form.  In  addition,  some  were  sent  an  advance  letter  with  a  straight- 
forward appeal  for  cooperation,  whereas  the  others  were  sent  an  advance 
letter  asking  for  the  sample  member's  advice  and  suggestions  on  the  data- 
collection  procedure.  In  contrast  to  Study  P2,  it  was  not  feasible  to  stratify 
accounts  by  size  in  the  sample  selection  process. 

Response  and  Nonresponse  Errors83 

The  extent  to  which  accounts  were  reported  and  the  correspondence 
between  reported  and  actual  balances,  by  type  of  form  used  and  by  study, 
are  shown  in  Table  31.  Overall,  nonreporting  of  accounts  in  the  urban 
study  was  substantial,  accounting  for  roughly  40  percent  of  all  of  the 
validated  accounts.  Nonreporting  was  higher  on  the  change  form  than 
on  the  holdings  form,  47  percent  versus  35  percent,  respectively,  which 
conforms  to  previous  findings.  Moreover,  column  4  of  Table  31  indi- 
cates that  the  average  size  of  these  nonreported  accounts  exceeded  by  a 
substantial  margin  the  average  size  of  the  accounts  which  were  reported 
and  for  which  balances  were  given. 

The  comparisons  obtained  from  the  holdings  form  with  regard  to  the 
reported  and  actual  sizes  of  accounts  are  particularly  interesting.  Espe- 
cially pertinent  are  the  following : 

(1)  The  average  nonrespondent  account  is  considerably  larger  than 
the  average  respondent  account  (column  4),  primarily  because  of  the 
large  account  balances  of  those  who  refused  to  give  figures. 

(2)  Accounts  for  which  balances  were  given  tend  to  be  considerably 
smaller  than  accounts  which  were  reported  but  for  which  the  balance 
was  refused. 


83  The  presentation  of  results  is  based  on  the  author's  article,  "The  Reliability 
of  Consumer  Surveys  of  Financial  Holdings:  Time  Deposits,"  Journal  of  the 
American  Statistical  Association,  Vol.  60,  No.  309   (March,  1965),  pp.   148-63. 
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Table  31.  Average  Size  of  Account,  by  Outcome 
of  Interview  Attempt  and  Form  Assigned 


Category 

(1) 


Total  sample 
Respondents.  .  .  . 

Nonreporters .  . 

Figure  refused . 

Figure  given. . , 
Nonrespondents. , 

Noncontacts. .  . 

Refusals 


Number 

of 
accounts81 

(2) 

Number 

of 
accounts'3 

(3) 

Average  size  of 
account1" 

Actual 

(4) 

Reported 

(5) 

Percentage 
of  devia- 
tion, re- 
ported from 
actual15'  ° 

(6) 


Study 

P2:    Urban 

196 

207 

$3,066 

$1,476 

158 

151 

2,663 

1,476 

75 

53 

2,985 

4 

7 

3,102 

79 

91 

2,469 

2,477 

38 

56 

4,153 

8 

9 

3,062d 

30 

47 

4,361e 

51.9 

44.6 

100.0 

-6.3 


Study  P3:   Farm 


Total  sample 
Respondents 

Nonreporters .  . 

Figure  refused . 

Figure  given. . , 
Nonrespondents. . 

Refusals 

Noncontacts. .  . 


35 

39f 

1,175 

390 

29 

32 

938 

390 

14 

7 

750 

0 

2 

6,588 

15 

23 

504 

543 

6 

7 

2,259 

4 

7 

2,259 

2 

66.8 
58.4 
100.0 
100.0 
-7.7 
100.0 
100.0 


a  Change  form. 

b  Holdings  form. 

c  Actual-reported  0 

Actual 
d  One  account  closed. 
e  Two  accounts  closed. 
f  Excludes  two  validated  accounts  on  holdings  form  closed  before  the  date  of  interview. 


(3)  The  average  size  of  reported  accounts  is  less  than  one-half  the 
actual  size  of  all  accounts.  In  other  words,  an  (unweighted)  estimate  of 
aggregate  time-deposit  holdings  for  this  sample  would  have  understated 
the  actual  total  by  roughly  52  percent. 

(4)  This  deviation  represents  the  accumulated  effects  of  nonreporting, 
of  large  accounts  for  which  balances  were  refused,  and  of  still  larger  ac- 
counts of  nonrespondents. 

(5)  Interestingly  enough,  the  average  size  of  accounts  for  which  bal- 
ances were  reported  corresponds  almost  exactly  with  the  true  figure. 

The  second  part  of  Table  31,  which  summarizes  the  extent  to  which 
farmers'  accounts  were  reported,  indicates  that  much  the  same  results 
were  obtained  in  this  study.  Nonreporting  was  less  frequent  than  in  Study 
P2,  occurring  on  roughly  one-third  of  these  interviews.  The  discrepancy 
between  the  incidence  of  nonreporting  on  the  change  form  and  its  in- 
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cidence  on  the  holdings  form  is  even  larger  than  before :  48  percent  versus 
22  percent,  respectively.  In  this  case,  the  difference  is  large  enough  to  be 
statistically  significant  at  the  .05  probability  level. 

Here,  as  was  the  case  with  other  validation  studies,  a  marked  tendency 
existed  for  people  not  to  report  holdings  when  asked  for  change  if  they 
had  no  or  little  change  to  report.  The  feeling  of  many  respondents  in 
such  cases  appeared  to  be  that  since  the  main  interest  was  apparently  in 
ascertaining  change,  there  was  no  reason  to  "bother"  the  interviewer  with 
additional  holdings  which  might  not  matter.  In  addition,  the  highly  per- 
sonal nature  of  the  request  for  holdings  tended  to  antagonize  so-called 
"marginal  respondents"  into  refusing  to  cooperate  altogether.  The  same 
type  of  people,  however,  when  asked  for  change  seemed  likely  to  remain 
in  the  sample  but  with  a  much  higher  tendency  to  omit  relevant  infor- 
mation. 

Data  obtained  from  the  holdings  form  indicate  that,  as  on  Study  P2, 
the  average  balance  in  these  accounts  was  underestimated  substantially. 
Although  they  were  relatively  few,  the  nonreporters  still  contributed  to 
this  underestimate.  In  addition,  the  balances  of  those  who  refused  this 
information  average  out  to  a  considerably  higher  figure  than  the  average 
obtained  from  the  respondents:  $6,588  versus  $543. 84  A  slight  offsetting 
tendency  is  apparent  in  the  overestimate  of  the  balances  that  were  re- 
ported. The  overall  result  is  that  the  sample  average  was  understated  by 
about  67  percent. 

Numerical  estimates  of  the  relative  importance  of  these  different 
sources  of  error  in  the  estimate  of  the  average-sized  balance  reveals  that 
virtually  all  the  error  on  Study  P2  is  attributable  almost  equally  to  in- 
accurate estimates  of  the  balances  of  the  nonrespondents  and  to  nonreport- 
ing.  The  error  allocation  is  as  follows : 

Study  P2      Study  P3 
Error  in  estimated  balances  of  the  nonrespondents ...      48 . 1  %         43 . 8% 

Error  due  to  nonreporters 50 .  7  17.6 

Inaccurate  estimates  of  those  refusing  balances 1.4  41.5 

Response  error —  .  2  —2.9 

100.0%       100.0% 

On  the  farm  study  (P3)  however,  nonreporting  is  seen  to  constitute 
a  relatively  small  source  of  error,  partly  because  nonreporting  was  some- 
what less  frequent  than  on  other  studies.  Also,  differences  between  the 
balances  given  by  the  respondents  and  the  actual  balances  of  the  non- 
reporters,  as  well  as  between  the  respondent  reports  and  the  balances  of 
the  nonrespondents,  were  relatively  larger  on  the  farm  study.    Although 

84  It  should  be  noted  that  there  were  only  two  refusals  in  this  category,  so  it 
is  not  clear  how  typical  such  a  discrepancy  may  be. 
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Table  32.   Percentage  Distribution  of  Validated  Accounts, 
by  Size  and  Type  of  Response,  Holdings  Form 


Size  of  account 

0) 


Actual  size 


Total 

(2) 


Nonre- 
spondents 

(3) 


Nonre- 
porters 

(4) 


Balance 
reported 

(5) 


Reported 
size 

(6) 


Stu 

dy  P2:    Urb 

an 

$0 

1.5 

5.5 

21.0 

11.5 

10.5 

6.5 

9.5 

12.0 

14.0 

8.0 

5.4 

3.6 

17.9 

12.5 

7.1 

3.6 

7.1 

12.5 

19.6 

10.7 

9A 
17.0 
13.2 

5.7 

7.5 
15.1 

9.4 
13.2 

9.4 

4.4 
25.3 

9.9 
15.4 

7.7 

7.7 
13.2 
11.0 

5.5 

36. 8a 

$1  to  $99 

2.1 

$100  to  $499 

15.3 

$500  to  $999 

6.9 

$1,000  to  $1,499 

$1,500  to  $1,999 

$2,000  to  $2,999 

$3,000  to  $4,999 

$5,000  to  $9,999 

$10,000  or  more 

11.1 
2.8 
6.9 
7.6 
5.6 
4.9 

Total 

100.0 

100.0 

99.9  . 

100.1 

100.0 

Base  number 

200 

56 

53 

91 

144 

Study  P3:    Farm 


$0 

$1  to  $99 

48.6 
10.8 
10.8 
10.8 
18.9 

28 '6 
14.3 

57^2 

57^1 

28^6 
14.3 

52.2 

13.0 

17.4 

8.7 

8.7 

23. 3a 
23.3 

$100  to  $499 

26.7 

$500  to  $999 

10.0 

$1,000  to  $1,999 

$2,000  or  more 

6.7 
10.0 

Total 

99.9 

100.1 

100.0 

100.0 

100.0 

37 

7 

7 

23 

30 

a  Includes  nonreported  accounts. 


the  nonrespondents,  and  particularly  the  respondents  who  refused  bal- 
ances, represented  relatively  small  parts  of  the  total  sample,  the  differences 
in  the  dollar  figures  were  so  large  that  these  two  factors  accounted  for 
the  principal  sources  of  error  in  this  study.  In  contrast,  response  errors 
were  very  small. 

Further  information  on  the  correspondence  between  the  actual  and 
reported  balances  for  the  urban  study  (P2)  is  provided  in  the  top  panel 
of  Table  32,  which  presents  size  distributions  of  these  accounts  by  different 
categories.  As  is  evident  from  this  table,  particularly  because  of  non- 
reporting,  the  size  distribution  of  reported  accounts,  shown  in  column  6, 
greatly  overestimates  the  frequency  of  zero  accounts.  Otherwise,  fairly 
close  correspondence  would  exist  between  the  distribution  of  reported 
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accounts  and  the  distribution  of  actual  balances  (column  2).  This  table 
also  serves  to  bring  out  the  heavy  preponderance  of  large  accounts  among 
the  nonrespondents  (column  3).  Over  30  percent  of  these  accounts  ex- 
ceeded $5,000,  which  was  true  of  only  one  out  of  every  five  other  accounts. 
The  table  also  shows  that  the  nonreported  accounts  were  concentrated 
at  both  ends  of  the  distribution.  Over  one-fourth  contained  less  than  $500 
each,  whereas  a  nearly  equal  proportion  contained  more  than  $5,000  each. 
The  effects  of  these  various  errors  on  the  distribution  of  reported  ac- 
counts by  size  in  the  farm  study  is  illustrated  in  the  lower  part  of  the 
table.  On  an  overall  basis,  the  sample  distribution  is  seen  to  overstate 
the  frequency  of  very  small  accounts  and  to  understate  the  frequency  of 
large  accounts  (column  2  versus  column  6).  Both  large  accounts  and 
very  small  accounts  are  seen  to  be  more  frequent  among  nonreporters  than 
among  reporters.  As  a  result,  the  frequency  of  large  accounts  is  under- 
stated and  the  frequency  of  small  accounts  is  overstated. 

Effect  on  Reliability 

The  following  results  are  obtained  in  estimating  the  reliability  of  the 

standard  error  of  the  mean : 

Study  P2       Study  P3 

k 5.1  4.5 

r -.62  -.92 

Apparent  standard  error  of  the  mean  (op) $     211  $113 

True  standard  error  of  the  mean  (\/MSE) $1,107  $590 

Error-free  standard  error  (oV) $     256  $289 

It  is  clear  that  the  errors  in  these  studies  introduced  substantial  dis- 
tortions in  the  estimates.  Without  having  accesss  to  the  validation  data, 
the  standard  error  of  the  mean  on  the  urban  study  would  have  been  esti- 
mated at  $211,  whereas  the  true  standard  error  of  the  mean  is  more  than 
five  times  that  figure.  Reflecting  this  fact  is  a  value  for  ft  of  5.1.  As  a 
result,  the  usual  symmetrical  95  percent  confidence  interval  has  a  prob- 
ability of  containing  the  true  parameter,  not  of  .95,  but  of  .001.  In  other 
words,  in  this  case  the  95  percent  confidence  interval  had  virtually  no 
likelihood  of  containing  the  true  value. 

The  intercorrelation  between  the  true  balances  and  the  errors  in  the 
respondent  reports  is  markedly  negative.  The  reason  for  this  is  the  high 
frequency  of  nonreporting,  which  serves  to  couple  xi  as  the  true  value 
with  —  Xi  as  the  amount  of  error.  Finally,  these  calculations  indicate  that 
had  these  nonsampling  errors  not  existed,  the  standard  error  of  the  mean 
would  have  been  roughly  one-fourth  of  the  true  standard  error  or,  as  it 
turns  out,  of  roughly  the  same  order  of  magnitude  as  the  apparent  stand- 
ard error  of  the  mean. 
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Table  33.  Tests  of  Association  Between  Selected 
Characteristics  and  Nonreporting,  Study  P2 


Characteristic 

Chi-square 

Degrees  of 
freedom 

Significance 
level 

Respondent  characteristics 

Education 

Occupation 

Age 

12.86 

11.82 

8.42 

8.67 

21.08 

4.87 

3 
4 
4 
4 
5 
5 

Less  than  .01 
.02 
.08 
.07 

Income  of  savings  unit 

Size  of  account 

Less  than  .01 
.44 

Value  of  t 

Cooperativeness:    objective  measures 

Use  of  records 

3.30 
2.30 

4.10 
3.91 

Less  than  .01 

Refusal  of  information 

Cooperativeness:   interviewer  ratings 

Accuracy  of  information 

.02 
Less  than  .01 

Completeness  in  reporting 

Less  than  .01 

Questionnaire  approach 

Holdings  versus  change 

Regular  letter  versus  evaluation  letter. .  . 

Forms  in  interview  versus  no  forms 

All  assets  versus  some  assets 

2. 
1. 
1. 

43 
28 
32 
23 

.01 
.20 
.19 
.82 

On  the  farm  study,  the  standard  error  of  the  mean  computed  in  the 
usual  way  is  also  much  too  low,  again  because  of  the  very  high  negative 
correlation  between  the  true  balances  and  the  errors,  so  that  response 
errors  serve  to  reduce  sharply  the  apparent  sampling  variance.85  As  before, 
the  true  standard  error  of  the  mean  turns  out  to  be  more  than  five  times  the 
size  of  the  apparent  standard  error.  Because  of  this  phenomenon  and 
the  large  bias  in  the  estimate  of  the  mean,  a  high  value  of  k  is  obtained 
again,  namely,  4.5.  In  this  case,  the  usual  symmetrical  .95  confidence 
interval,  based  on  the  apparent  standard  error,  has  a  probability  of  con- 
taining the  true  population  mean  of  roughly  .005. 

Determinants  of  Response  Error 

Four  broad  types  of  variables  were  tested  for  association  with  the  fre- 
quency of  nonreporting :  characteristics  of  the  savings  unit,  objective 
measures  of  respondent  cooperativeness,  subjective  rating  of  respondent 
cooperativeness  (based  on  interviewer  reports),  and  the  questionnaire 
approach.  The  results  of  these  tests  for  the  urban  study,  shown  in  Table 
33,  indicate  that  most  of  the  variables  in  each  category  were  related  sig- 

85  This  high  correlation  was  due  principally  to  five  accounts  with  balances 
between  $1,000  and  $7,000,  three  of  which  were  not  reported  and  two  of  which 
were  understated  by  about  90  percent.  If  these  accounts  were  eliminated,  the 
correlation  would  be  virtually  zero. 
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nificantly  to  nonreporting.  With  the  exception  of  the  size  of  the  account, 
all  of  the  respondent  characteristics  were  related  significantly  to  the  fre- 
quency of  nonreporting,  or  were  almost  so.86  Furthermore,  in  all  instances 
the  relationship  is  in  accord  with  what  might  be  expected.  Thus,  the  fre- 
quency of  nonreporting  is  higher  among  those  with  less  education,  those 
in  a  laboring  occupation  or  retired,  older  people,  those  with  lower  incomes, 
and  those  who  are  in  larger-sized  savings  units. 

Highly  significant  relationships  are  also  apparent  between  nonreport- 
ing and  the  various  measures  of  cooperativeness.  As  would  be  expected, 
frequency  of  reporting  increases  with  use  of  records  and  with  a  lower  in- 
cidence of  refusals.  Particularly  noteworthy  is  the  fact  that  the  inter- 
viewers appear  to  be  able  to  detect  nonreporting  (although  not  aware  that 
they  are  doing  so)  much  more  frequently  than  would  be  expected  as  a 
result  of  chance.  The  results  therefore  suggest  that  interviewer  ratings 
might  be  of  assistance  in  dealing  with  this  problem.  In  addition,  Table 
33  brings  out  once  more  the  superiority  of  the  holdings  form  over  the 
change  form. 

As  was  noted  previously,  a  substantially  higher  incidence  of  reporting 
is  obtained  with  the  holdings  form  than  with  the  change  form,  the  dif- 
ference being  statistically  significant.  With  the  other  experiment,  however, 
virtually  no  effect  is  apparent,  the  regular  letter  yielding,  if  anything,  a 
slightly  lower  frequency  of  nonreporting  than  the  evaluation  letter. 

Table  34  presents  the  results  of  significance  tests  of  the  extent  to  which 
reporting  of  an  account  on  the  farm  study  was  associated  with  the  same 
four  general  categories  of  variables  tested  in  Study  P2.  As  is  evident  from 
this  table,  in  Study  P3  many  fewer  characteristics  are  related  significantly 
to  reporting  of  the  account  than  in  Study  P2,  perhaps  because  of  the 
smaller  sample  sizes  involved  or  perhaps  because  of  the  different  nature 
of  the  sample,  focusing,  as  it  does,  on  a  farm  area. 

Of  the  respondent  characteristics,  only  income,  farm  tenancy,  and 
operator  status  are  statistically  significant,  with  tenants  reporting  the  ac- 
count more  frequently  than  owners,  and  operators  reporting  the  account 
more  frequently  than  nonoperators.  Although  not  significant,  there  is 
some  indication  that  education,  age,  and  size  of  savings  unit  may  have 
some  effect,  the  nature  of  the  relationships  being  the  same  as  before. 

Of  the  measures  of  cooperativeness,  only  use  of  records  is  statistically 
significant,  with  those  using  records  reporting  accounts  far  more  frequently 
than  those  who  did  not.  It  is  interesting  to  note  that  a  significant  relation- 
ship is  obtained  for  interviewer  ratings  of  completeness  in  reporting,  but 
not  for  interviewer  ratings  of  the  accuracy  of  information.    Clearly,  these 

88  Strictly  speaking,  size  of  account  is  not  a  characteristic  of  the  respondent, 
although  it  is  classified  here  under  this  heading  for  convenience. 


116     The  Reliability  of  Consumer  Reports  of  Financial  Assets  and  Debts 


Table  34.  Tests  of  Association  Between  Selected 
Characteristics  and  Nonreporting,  Study  P3 


Characteristic 


Respondent  characteristics 

Education 

Age 

Size  of  savings  unit 

Income  of  savings  unit 

Owner  versus  tenant 

Operator  versus  nonoperator .  .  .  . 
Cooperativeness:    objective  measures 

Use  of  records 

Refusal  of  information 

Cooperativeness:    interviewer  ratings 

Accuracy  of  information 

Completeness  in  reporting 

Questionnaire  approach 

Holdings  versus  change 

Regular  versus  evaluation  letter.  . 

a  Value  of  t  rather  than  chi-square. 


Chi-square 


3.56 
4.77 
4.27 
5.95 
8.54 
13.87a 


.66 
7.33a 

2.22a 
.50a 


Degrees  of 
freedom 


Significance 
level 


.18 
.19 
.12 
.05 
.02 
Less  than  .01 

.03 
.25 

.72 
Less  than  .01 

.03 
.62 


two  questions  register  different  things  in  the  minds  of  interviewers  —  in 
this  instance,  the  dimension  of  completeness  was  the  more  pertinent. 


The  Netherlands  Reliability  Survey87 
Background 

In  1958,  the  Economic-Statistical  Department  of  the  Post  Office  Sav- 
ings Bank  of  the  Netherlands  undertook  an  experimental  survey  to  investi- 
gate ( 1 )  the  reliability  with  which  individuals  in  the  Netherlands  were 
reporting  savings  information  in  personal  interviews  and  (2)  the  factors 
that  might  be  related  to  the  reliability  of  this  information.  Previous  com- 
parison of  external  information  with  data  from  savings  surveys  conducted 
in  1950,  1953,  and  1955  by  the  Post  Office  Savings  Bank  had  raised  doubt 
concerning  the  reliability  of  the  interview  data. 

The  1958  survey  was  undertaken  using  the  accounts  in  the  Post  Office 
Savings  Bank  as  the  validation  source.  The  frame  consisted  of  account 
holders  in  the  Post  Office  Savings  Bank  living  in  four  urban  areas  of  the 
country  and  having  a  balance  of  more  than  10  guilders  as  of  January  1, 
1958. 88  To  avoid  ambiguities  in  response,  the  survey  was  limited  to  ac- 
count holders  with  only  one  account.  It  was  further  restricted  to  people 
who  were  20  years  or  more  of  age  and  who  had  opened  their  own  account. 

87  The  material  in  this  selection  is  based  on  W.  Horn,  "Reliability  Survey:  A 
Survey  on  the  Reliability  of  Response  to  an  Interview  Survey,"  Het  PTT-bedrijf, 
Vol.  10,  No.  3  (October,  1960),  pp.  105-56. 

88  A  guilder  as  of  that  date  was  equivalent  to  26  cents. 
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A  stratified  sample  was  selected  from  this  population,  with  stratifica- 
tion proportionate  to  the  percentage  of  total  balances  in  each  of  three  size 
classes  of  accounts:  10-1,000  guilders;  1,000-2,500  guilders;  and  2,500 
guilders  or  more.  As  a  result,  there  was  heavy  overrepresentation  of  the 
larger  accounts,  with  more  than  one-third  of  the  sample  members  having 
accounts  over  2,500  guilders,  although  less  than  5  percent  of  the  popula- 
tion had  accounts  of  that  size. 

The  questionnaire  covered  a  variety  of  topics  relating  to  savings  prac- 
tices and  attitudes.  Questions  were  asked  about  the  general  economic 
situation,  current  income,  income  expectations,  and  opinions  about  the 
need  and  means  of  saving,  as  well  as  questions  about  holdings  in  savings 
institutions  (including  the  Post  Office  Savings  Bank),  checking  accounts, 
ownership  of  a  business,  and  a  variety  of  economic  and  demographic 
characteristics.  Respondents  were  not  told  that  the  survey  was  being  con- 
ducted by  the  Post  Office  Savings  Bank. 

Three  experiments  were  incorporated  into  this  survey  design,  as 
follows : 

(1)  The  relative  efficiency  of  male  versus  female  interviewers.  For 
this  purpose,  the  field  staff  consisted  of  32  male  interviewers  and  32  female 
interviewers. 

(2)  The  effectiveness  of  a  combination  of  oral  and  written  inter- 
viewer instructions  versus  the  use  of  only  written  instructions. 

(3)  The  mailing  of  advance  letters  to  the  respondent  versus  not  send- 
ing any  advance  notification. 

As  was  done  in  the  various  studies  in  the  United  States,  these  experi- 
ments were  combined  with  each  other  in  a  factorial  design  with 
interviewer-interviewing  approaches  allocated  systematically  among  the 
sample  members. 

Operating  Results 

Interviews  were  obtained  with  nearly  80  percent  of  the  sample  mem- 
bers. About  10  percent  of  the  sample  refused  to  be  interviewed,  and 
another  12  percent  could  not  be  contacted  for  one  reason  or  another.  Most 
of  the  latter  (about  85  percent  of  the  12  percent)  were  not  at  home  or 
were  too  ill  to  be  contacted;  the  rest  represented  people  who  had  moved 
or  died. 

Analysis  of  the  nonrespondents  on  the  basis  of  institutional  informa- 
tion indicated  that  the  noncontact  rate  was  much  higher  among  the 
youngest  (20-34)  and  the  oldest  (65  and  over)  age  groups  than  among 
others.  The  high  rate  in  the  former  case  was  due  primarily  to  people 
either  not  being  at  home  or  having  moved,  whereas  the  high  noncontact 
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rate  in  the  older  age  groups  was  due  to  the  much  higher  incidence  of  sick- 
ness and  death. 

No  relationship  was  found  between  the  noncontact  rate  and  the  size  of 
the  account.  Such  a  relationship  was  found,  however,  between  the  re- 
fusals —  the  refusal  rate  tended  to  rise  as  the  size  of  the  account  increased 
—  and  the  reporting  of  balances.  Thus,  only  7  percent  of  the  sample 
members  in  the  lowest  account  group  refused  to  state  a  balance,  whereas 
this  rate  rose  to  almost  25  percent  of  those  with  accounts  of  2,500  guilders 
or  more. 

Two  other  factors  were  found  to  influence  the  refusal  rate.  One  factor 
was  sex,  with  relatively  fewer  refusals  received  from  men  than  from 
women.  In  addition,  a  strong  upward  tendency  was  found  for  the  rate 
of  refusal  to  increase  with  age.  The  refusal  rate  varied  from  5.6  percent 
of  those  aged  20-34  to  14.5  percent  of  those  who  were  65  years  of  age  or 
more. 

Although  almost  80  percent  of  the  sample  members  were  interviewed, 
complete  information  on  the  savings  account  was  supplied  by  little  more 
than  half  of  the  sample.  About  5  percent  of  the  sample  members  denied 
having  an  account  and  an  additional  19  percent,  or  almost  one-fourth  of 
those  interviewed,  mentioned  the  existence  of  the  account  but  refused  to 
give  the  current  balance.  The  tendency  not  to  report  an  account  was 
much  higher  for  the  smallest  accounts  than  for  any  others.  This  phenom- 
enon suggests  that,  to  some  extent,  nonreporting  may  have  been  acci- 
dental, reflecting  poor  memory.  Support  for  this  inference  was  found  in 
the  fact  that  nonreporting  in  relation  to  the  total  number  of  accounts  rose 
substantially  as  the  interval  between  the  date  of  the  interview  and  the 
date  of  the  last  transaction  increased.89 

Nonreporting  was  also  related  significantly  to  age  and  to  education, 
but  not  to  sex.  Older  people  were  more  inclined  not  to  report  an  account 
than  those  in  other  age  groups,  as  were  people  with  only  an  elementary 
school  education.  There  was  some  tendency  for  low-income  people  not  to 
report  more  frequently  than  others,  but  this  may  reflect  primarily  inter- 
acting effects  of  education  and  (low)  size  of  account. 

Interestingly  enough,  some  relationship  was  found  between  nonreport- 
ing and  interviewer  characteristics,  as  well  as  between  nonreporting  and 
respondent-interviewer  interactions.  Thus,  older  interviewers  were  found 
to  receive  significantly  more  denials  of  account  ownership  than  younger 
interviewers.  In  addition,  nonreporting  was  significantly  lower  in  in- 
stances involving  female  interviewers  and  male  respondents  than  with  any 
other  combination. 


89  Actually  the  argument  is  not  complete,  because  Horn  fails  to  demonstrate 
whether  or  not  the  size  of  the  account  was  inversely  related  to  the  interval  between 
the  last  transaction  and  the  date  of  the  interview. 
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An  age  interaction  effect  was  also  detected,  the  frequency  of  nonre- 
porting  tending  to  rise  as  the  age  of  the  interviewer  relative  to  the  age  of 
the  respondent  decreased. 

Errors  in  Balance  Reports 

A  general  summary  of  the  errors  in  estimates  of  the  sizes  of  the  ac- 
counts is  presented  in  Table  35,  for  the  total  sample  as  well  as  for  four 
different  size-of-account  categories.  The  data  in  the  "total"  column  are 
weighted  by  the  reciprocals  of  the  sampling  fractions  used  for  the  different 
size-of-account  categories,  to  present  an  unbiased  estimate  of  the  popula- 
tion totals. 

On  balance,  the  data  suggest  (lines  1-6)  that  the  average  account  of 
the  nonrespondents  was  somewhat  larger  than  that  of  the  respondents  and 
that  those  who  refused  to  give  balances  had  accounts  much  larger  than 
the  average.  At  the  same  time,  nonreporters,  as  well  as  those  who  reported 
both  an  account  and  the  balance  in  the  account,  had  considerably  smaller 
accounts  than  the  average. 

The  data  further  suggest  that  these  patterns  are  not  always  true  for  all 
size-of-account  categories.  Thus,  in  the  lower  account  categories  nonre- 
spondents tended  to  have  smaller  accounts  than  respondents,  and  it  was 
only  for  accounts  of  2,500  guilders  or  more  that  nonrespondents'  accounts 
exceeded  those  of  respondents.  In  a  similar  manner,  only  among  the 
largest  accounts  was  the  average  for  those  who  gave  a  balance  less  than 
for  other  sample  members.  Also,  the  primary  reason  for  the  very  low  aver- 
age size  of  accounts  for  the  nonreporters  was  a  large  number  of  extremely 
low  accounts;  at  the  upper  levels,  the  average  account  of  this  group  ex- 
ceeded the  average  for  the  sample  as  a  whole.  Only  for  those  who  refused 
to  give  balances  did  the  average  account  exceed  the  overall  average  for  all 
size-of-account  categories. 

To  some  extent  these  results  are  not  too  surprising,  since  undoubtedly 
various  motives  interacted  with  the  tendency  to  report  accurately  about 
current  sizes  of  accounts.  Thus,  with  very  small  accounts,  memory  error 
was  more  likely  to  play  a  major  role.  As  the  size  of  account  increased, 
however,  interaction  between  this  factor  and  any  suspicion  or  fear  that 
might  have  existed  on  the  part  of  the  respondent  would  have  been  mani- 
fested either  in  the  form  of  nonreporting  or  in  refusal  to  give  the  balance. 

The  effect  of  these  different  errors  on  estimates  of  the  average  balance 
is  shown  in  lines  7-10  of  Table  35.  As  is  evident  from  line  9,  the  response 
error  is  relatively  small,  an  overstatement  of  5  percent.  However,  this 
overstatement  reflects  the  cancellation  of  substantial  opposing  errors,  par- 
ticularly an  overstatement  of  71  percent  in  the  estimate  of  the  average 
account  in  the  lowest  category.   Fortunately,  from  this  respect  the  largest 
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accounts  were  understated  by  10  percent  which,  because  of  the  size  of 
these  accounts,  served  to  almost  offset  the  appreciable  overstatement  of 
the  smallest  accounts. 

The  best  estimates  of  the  average  size  holdings  of  the  total  sample 
from  the  interview  data  are  shown  in  line  8.  The  overall  average  is  seen 
to  be  understated  by  14  percent  (line  10),  again  representing  the  result 
of  cancellation  of  much  larger  opposing  errors.  The  average  size  of  the 
smallest  accounts  is  overstated  by  81  percent,  which  was  more  than  offset 
by  a  21  percent  understatement  of  the  largest  accounts  and  by  an  under- 
statement of  10  percent  of  accounts  in  the  next  largest  category. 

The  error  in  these  estimates  is  allocated  among  four  different  causes  in 
the  last  six  lines  of  the  table,  as  was  done  in  the  previous  studies.  Overall, 
the  results  of  this  error  allocation  are  seen  to  correspond  fairly  closely  with 
those  obtained  earlier.  Nonreporting  is  the  principal  cause  of  error,  lead- 
ing, with  errors  in  the  estimates  of  refused  balances,  to  understatement  of 
the  true  figure.  Response  errors  are  in  the  direction  of  overestimation  and 
hence  help  to  offset  to  some  extent  the  other  causes  of  error.  However, 
strikingly  different  error  patterns  are  observed  by  size-of-account  cate- 
gories. This  is  especially  true  of  the  smallest  accounts,  where  response 
errors  represent  most  of  the  error  in  the  average  size  of  the  account, 
whereas  errors  due  to  nonreporting  are  negligible.  In  this  particular  case, 
the  average  size  of  the  accounts  not  reported  was  so  small  that  even  the 
relatively  high  frequency  of  such  accounts  exerted  little  effect  on  the 
overall  estimate. 

The  error  in  the  estimate  of  the  average  size  of  accounts  in  the  100- 
1,000  guilder  range  is  particularly  interesting.  Although  the  overall  error 
is  almost  negligible  —  an  underestimate  of  2  percent  —  it  is  seen  to  be 
the  result  of  the  cancellation  of  relatively  large  error  tendencies  in  oppos- 
ing directions.  Error  due  to  nonreporting  was  almost  twice  the  size  of  the 
overall  error,  whereas  the  error  in  the  estimate  of  balances  refused  almost 
equalled  the  overall  error.  At  the  same  time,  the  response  error  in  this 
case  was  both  relatively  substantial  and  in  the  opposite  direction  to  the 
other  errors,  so  the  resulting  cancellation  of  errors  produced  a  close  overall 
estimate. 

Determinants  of  Errors  in  Balances 

A  number  of  factors  were  found  to  influence  the  reliability  of  the 
report  of  account  balances.  Among  these  were  the  following : 

(1)  Willingness  to  consult  bankbooks.  If  a  respondent  did  not  consult 
his  bankbook  in  giving  the  balance  figure  or  did  not  refuse  to  answer,  he 
was  told:  "We  should  like  to  know  exact  amounts.  Are  you  willing  to 
look  (at  your  account)  and  tell  us  exactly  how  much  you  have?"    Those 
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who  had  consulted  their  passbooks  previously,  or  did  so  in  response  to 
this  question,  gave  considerably  more  accurate  information  than  those 
who  did  not  consult  bankbooks.  In  the  former  case,  the  overall  error  was 
almost  negligible,  compared  with  underestimates  of  the  order  of  20  per- 
cent for  those  who  gave  a  figure  but  refused  to  consult  their  bankbook. 

(2)  Size  of  account.  Small  accounts  tended  to  be  overstated  and  large 
accounts  to  be  understated.  When  the  balance  reports  were  arranged  in 
descending  order  and  combined  into  groups  of  100,  a  linear  relationship 
was  obtained  between  the  error  and  the  size  of  the  balance,  with  a  corre- 
lation coefficient  for  the  17  observations  of  —.96. 

(3)  Income.  A  much  looser  relationship  was  apparent  between  error 
and  level  of  income,  with  some  tendency  for  those  at  the  upper-income 
levels  to  overstate  the  actual  size  of  the  account  and  those  at  lower-income 
levels  to  understate  the  size  of  the  account.  In  addition,  income  was  found 
to  affect  the  relationship  between  error  and  size  of  account,  the  zero  point 
for  error  tending  to  rise  with  income  level.  A  multiple  correlation  coeffi- 
cient of  .73  resulted  when  error  was  regressed  on  income  level  and  on  size 
of  the  account,  after  combining  the  observations  into  groups  of  10  by  size 
of  account  within  income  level. 

(4)  Age.  Account  balances  reported  by  older  people  were  more  likely 
to  be  in  error  than  those  reported  by  younger  people.  As  Horn  notes,  this 
phenomenon  might  be  explained  for  the  most  part  by  the  larger  accounts 
of  these  people  and  by  their  greater  unwillingness  to  consult  bankbooks. 

(5)  Use  of  an  introductory  letter.  Recipients  of  an  introductory  letter 
were  more  likely  to  consult  bankbooks  than  those  not  sent  such  a  letter 
(65  percent  versus  56  percent,  respectively) . 

(6)  Interval  between  date  of  interview  and  date  for  which  balance 
was  requested.  In  the  interview,  balances  were  requested  for  two  dates  — 
the  date  of  the  interview  (some  time  during  September-October,  1958) 
and  January  1,  1958.  Errors  in  the  balance  given  for  the  earlier  date  were 
much  more  frequent  than  errors  for  the  later  date.  Oddly  enough,  the 
biggest  differential  occurred  among  those  who  consulted  bankbooks,  30 
percent  of  whom  erred  about  the  balance  on  the  interview  date  and  53 
percent  of  whom  gave  the  incorrect  balance  on  January  1,  1958.  (How- 
ever, apparently  most  of  these  errors  were  relatively  small  —  between  2 
and  100  guilders.) 

Estimates  of  Saving 

Since  the  questionnaire  sought  balance  data  for  two  different  dates, 
estimates  of  respondents'  saving  between  these  dates  could  be  computed. 
Comparison  of  these  computed  estimates  with  the  actual  figures  indicated 
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substantial  underestimation  of  actual  saving,  the  figures  being  as  follows 
(in  guilders)  :90 

Average  balance  on  Actual  Computed        Difference 

Interview  date 731  630  101 

January  1,  1958 645  617  28 

Saving 86  13  73 

Since  the  degree  of  understatement  was  much  larger  for  the  balance 
on  the  interview  date  than  for  the  balance  as  of  January  1,  1958,  the 
computed  average  saving  was  very  much  lower  than  the  actual  figure. 

Comparison  of  actual  saving  with  computed  saving  for  individual 
respondents  indicated  that  the  absolute  amount  of  saving  was  understated 
at  both  ends  of  the  distribution:  the  greater  the  amount  of  saving,  or  of 
dissaving,  the  larger  the  extent  of  understatement.  This  relationship  was 
measured  by  means  of  a  regression  line  of  average  error  on  average  actual 
saving,  combining  the  observations  into  groups  of  25  by  amount  of  saving. 
The  relationship  was  negative,  with  a  correlation  coefficient  of  .94. 

Comparative  Evaluation 

Overall,  these  time-deposit  validation  studies  point  to  the  existence  of 
substantial  errors  in  such  data  when  obtained  by  consumer  surveys.  Thus, 
we  find  that : 

( 1 )  Sample  averages  tend  to  understate  substantially  the  true  popula- 
tion figures. 

(2)  As  a  general  rule,  the  principal  cause  of  these  underestimates  is 
the  nonreporting  of  respondents.  A  second  major  factor  appears  to  be 
underestimates  of  the  holdings  of  the  nonrespondents. 

(3)  Response  errors  are  generally  small.  As  was  true  in  the  case  of 
debt,  there  is  some  tendency  toward  overstatement,  but  this  tendency  is 
slight. 

(4)  A  finding  that  runs  counter  to  the  corresponding  one  for  debt  is 
the  overstatement  of  the  frequency  of  small  savings  accounts.  There  is 
also  some  tendency  for  the  frequency  of  large  savings  accounts  to  be  un- 
derstated, particularly  if  nonreporters  are  included  in  the  size  distribu- 
tions. This  tendency  toward  the  so-called  average-man  effect  —  by  which 
it  is  meant  that  people  tend  to  bias  their  replies  in  accordance  with  their 
visualization  of  the  actions  of  the  average  man  —  has  been  found  to  exist 
also  in  a  number  of  other  instances.91 


90  Horn,  op.  cit.,p.  141. 

91  See  E.  S.  Maynes,  "The  Anatomy  of  Response  Errors:  Sample  Survey  Data 
on  Saving,"  Journal  of  Marketing  Research,  Vol.  2,  No.  4  (November,  1965), 
pp.  378-87. 
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(5)  The  tenuous  generalization  in  the  case  of  debt  that  nonrespondents 
hold  more  than  respondents  do  receives  strong  support  in  the  case  of 
savings  accounts.  In  some  of  these  studies,  the  average  balances  of  the 
nonrespondents  were  several  times  as  large  as  those  of  the  respondents. 

(6)  The  bias  in  the  estimates  of  the  means  leads  to  substantial  errors 
in  the  estimates  of  confidence  intervals.  The  confidence  intervals  obtained 
from  the  sample  observations  can  understate  tremendously  the  true  size 
of  the  confidence  interval,  stemming  from  the  dominance  of  the  square  of 
the  bias  in  the  expression  for  the  true  variance  of  the  mean.  A  strong 
negative  correlation  may  also  exist  between  the  true  value  and  the  re- 
sponse error,  due  to  frequent  nonreporting,  and  this  may  reduce  the 
apparent  standard  error  of  the  mean  to  a  spuriously  low  level. 

As  was  true  in  the  case  of  debt,  accuracy  of  report  was  affected  sub- 
stantially by  a  number  of  different  factors.  These  were  found  to  include 
characteristics  of  the  savings  unit,  objective  measures  of  respondent  co- 
operativeness,  interviewer  ratings  of  respondent  cooperativeness,  and  the 
questionnaire  approach.  Again,  strong  evidence  is  provided  for  the 
greater  reliability  of  a  holdings  form  compared  with  that  of  a  change  form 
and  for  using  an  approach  whereby  the  respondent  participates  in  the 
evaluation  of  the  interview.  As  before,  both  alternatives  yield  lower 
response,  but  nevertheless  there  is  no  question  regarding  the  superiority  of 
the  resulting  data. 

The  relationships  between  accuracy  of  reporting  and  respondent  char- 
acteristics were  essentially  the  same  as  in  the  case  of  debt,  with  one  ex- 
ception :  in  most  of  these  studies,  reporting  of  holdings  was  not  associated 
especially  closely  with  the  level  of  income,  and  where  such  an  association 
was  present,  higher  accuracy  was  generally  obtained  from  the  upper- 
income  levels.  Particularly  interesting  is  the  strong  association  in  some 
studies  between  the  reporting  of  holdings  and  interviewer  ratings  of  re- 
spondent cooperativeness.  These  results  suggest  that  information  supplied 
by  properly  trained  interviewers  may  be  of  considerable  aid  in  detecting 
response  errors. 


V.  VALIDATION  STUDIES:  OTHER  ASSETS 

This  chapter  completes  our  coverage  of  one-time  validation  studies, 
presenting  validation  results  relating  to  life  insurance,  demand  deposits, 
and  farm  assets.  Some  of  these  are  primary  validation  studies;  others  are 
secondary  validations. 

Life  Insurance 

Ownership  of  life  insurance  policies  was  checked  on  the  three  panel 
studies  of  the  Consumer  Savings  Project  —  PI,  P2,  and  P3.  However, 
the  results  do  not  possess  the  reliability  of  many  of  the  previously  discussed 
analyses  because  in  all  instances  this  asset  was  a  secondary  validation 
source.  Holdings  could  be  checked  only  for  companies  with  which  insur- 
ance was  reported,  and  not  for  all  of  these.  Therefore,  the  full  extent  of 
nonreporting  of  life  insurance  holdings  could  not  be  ascertained. 

In  contrast  to  the  other  asset  validations,  life  insurance  in  each  panel 
operation  was  validated  only  on  a  one-time  basis  on  a  wave  of  interviews 
later  than  the  first.  This  was  done  in  part  because  of  the  difficulty  of 
validating  these  data,  which  combined  with  the  relatively  static  nature 
of  insurance  holdings,  at  least  in  the  short  run,  did  not  make  more  than 
one  check  per  year  feasible.  Partly  too,  experimentation  with  different 
questionnaire  approaches  meant  that  information  on  the  holdings  of  all 
panel  members  could  not  be  available  before  the  second  or  third  wave, 
depending  on  the  particular  experiment.  Because  of  these  limitations,  the 
validity  of  the  results  is  greatly  limited,  particularly  with  regard  to  non- 
reporting.  Their  principal  value  lies  in  highlighting  the  nature  and  scope 
of  response  errors. 

Data  Collection 

In  Study  PI,  it  will  be  recalled  (p.  77),  one  questionnaire  variant  was 
not  to  request  dollar  amounts  until  the  second  wave  of  half  those  inter- 
viewed with  a  structured  form  and  half  those  interviewed  with  an  unstruc- 
tured form.  Hence,  information  on  the  life  insurance  ownership  of  all  the 
panel  members  in  Study  PI  was  not  available  until  Wave  2.  In  addition, 
by  the  time  the  necessary  validation  arrangements  had  been  made,  the 
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Wave  3  interviews  had  already  been  conducted,  so  this  Wave  3  infor- 
mation served  as  the  basis  for  the  validation.  Also  in  Studies  P2  and  P3, 
full  details  about  life  insurance  ownership  of  the  sample  members  was  not 
obtained  until  Wave  3,  again  because  of  the  experimental  aspects.  Hence, 
for  these  studies  as  well,  the  Wave  3  information  served  as  the  basis  for 
validation. 

In  Study  PI,  672  life  insurance  policies  reported  by  173  savings  units 
were  submitted  for  validation  on  face  value  and  on  premium  payment; 
term  insurance  was  excluded  —  it  is  not  a  form  of  saving.  The  compa- 
rable figures  for  Studies  P2  and  P3  are  634  policies  of  200  savings  units 
and  757  policies  of  278  savings  units,  respectively.  Since  the  results  ob- 
tained on  both  bases  were  essentially  the  same,  results  are  presented  only 
in  terms  of  face  value;  the  principal  difference  was  a  tendency  for  fewer 
"don't  knows"  or  "no  answers"  to  be  received  on  face  value  than  on 
premium  payment. 

The  type  of  validation  information  obtained  is  shown  by  the  following 

tabulation : 

Study  PI       Study  P2       Study  P3 

Policy  verified  and  face  value  supplied ....  376  259  267 

Policy  verified  but  no  face  value  given ....  7  2  5 

Policy  could  not  be  located 269  322  51 

No  report  received 20  51  434 

Total 672  634  757 

In  other  words,  fully  useable  validation  information  was  returned  on 
little  more  than  half  (55  percent)  of  the  policies  submitted  in  Study  PI, 
about  40  percent  in  Study  P2,  and  about  35  percent  in  Study  P3.  By  far 
the  main  reason  for  this  low  rate  of  return  was  inability  to  locate  policies 
or  names  of  policyholders,  despite  the  fact  that  information  was  supplied 
on  the  name  of  the  respondent,  address,  age,  type  of  policy,  and  length 
of  time  held.  Among  the  contributing  factors  were  the  following: 

(1)  Given  names  of  wives  and  children  were  not  generally  obtained 
when  policies  were  in  their  name.  Particularly  awkward  was  the  situation 
when  (usually  unknown  to  the  interviewer)  a  policy  was  in  a  wife's 
maiden  name. 

(2)  Policies  whose  holders  had  common  names  were  difficult  to  val- 
idate, since  current  addresses  did  not  often  correspond  with  addresses  at 
the  time  the  policy  was  taken  out. 

(3)  Most  companies  did  not  have  a  master  file  of  all  policyholders, 
but  had  instead  a  number  of  files  for  different  types  of  insurance.  For 
some  types  of  insurance,  such  as  debt  insurance,  the  only  file  of  policy- 
holders might  be  in  local  agency  offices.  For  industrial  insurance,  records 
were  kept  by  policy  number,  so  that  names  could  not  readily  be  located. 
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(4)  Considerable  confusion  was  apparent  among  respondents  on  name 
of  company  and  type  of  policy,  the  former  clearly  compounded  by  the 
widespread  similarity  of  company  names  in  the  industry.  Except  where 
records  were  used,  wrong  names  of  companies  seem  to  have  been  a 
common  occurrence.  In  this  connection,  it  is  pertinent  to  note  that  none 
of  the  policies  reported  by  41  of  173  savings  units  in  Study  PI  could  be 
located  by  the  company  named  by  the  respondent. 

(5)  Some  companies  refused  to  supply  the  requested  information,  al- 
though such  instances  were  relatively  few. 

Study  of  respondent  reports  on  the  policies  that  could  not  be  validated 
suggests  that,  on  the  average,  they  were  considerably  smaller  than  policies 
that  were  validated:  the  average  face  value  of  these  policies,  compared 
with  the  average  face  value  of  the  validated  policies,  was  as  follows: 

Study  PI       Study  P2       Study  P3 

Not  validated $3,123         $1,866         $2,427 

Validated    8,127  3,636  2,915 

Many  of  these  nonvalidated  policies  appear  to  have  been  industrial  or 
group  policies,  and  they  were  also  much  more  likely  to  have  been  taken 
out  on  a  member  of  the  savings  unit  other  than  the  main  wage  earner. 
To  judge  by  the  fact  that  records  were  used  about  as  often  with  these 
policies  as  with  others,  plus  the  fact  that  cooperativeness  was  about  the 
same,  the  only  basis  for  expecting  a  different  pattern  of  response  error  for 
these  policies  than  for  those  validated  would  be  some  relationship  between 
accuracy  of  response  and  size  of  policy.  This  question  will  be  considered 
shortly. 

The  smaller  size  of  the  nonvalidated  policies  also  may  explain  the 
greater  frequency  of  such  policies  in  the  later  studies.  The  income  status 
of  the  members  of  Studies  P2  and  P3  was  generally  considerably  lower 
than  that  of  the  members  of  Study  PI,  and  this  was  reflected  in  the  much 
higher  frequency  of  small  policies.  Thus,  the  average  face  value  of  policies 
that  could  not  be  located  by  the  institution  (and  for  which  dollar  figures 
were  reported  by  the  respondent)  in  Study  P2  was  $1,866,  in  contrast 
to  an  average  face  value  of  $3,123  for  the  corresponding  policies  in  Study 
PI.  Furthermore,  40  percent  of  the  policies  that  could  not  be  traced  in 
Study  P2  had  a  (reported)  face  value  of  less  than  $1,000,  compared  with 
12  percent  of  those  that  were  traced.  Examination  of  the  interview  data 
and  discussions  with  company  representatives  tended  to  confirm  the  sus- 
picion that  a  much  higher  proportion  of  policies  in  Study  P2  than  in 
Study  PI  were  either  group  policies  or  industrial  policies.92 

92  In  addition,  inability  to  induce  a  principal  company  in  Study  P3  to  supply 
validation  information  accounts  for  126  of  the  434  policies  for  which  no  report 
was  received.    No  comparable  situation  was  encountered  on  the  two  other  studies. 
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Magnitude  of  Error 

A  general  picture  of  the  response  errors  on  the  policies  that  were  val- 
idated is  provided  in  Table  36,  together  with  figures  on  the  extent  to 
which  additional  nonreported  policies  were  picked  up  in  the  validation. 
This  pickup  was  made  possible  because  information  was  sought  on  all 
policies  owned  by  the  respondent  with  the  particular  company,  not  just 
on  validated  policies.  For  reasons  noted  earlier,  this  pickup  serves  as  a 
lower  limit  estimate  of  the  extent  of  nonreporting  of  life  insurance  policies. 

As  is  evident  from  Table  36,  this  pickup  varied  from  57  policies  not 
mentioned  in  the  interviews  on  Study  PI  to  only  15  policies  on  Study  P3. 
These  represent  about  15  percent  of  all  validated  policies  on  Study  PI 
(17  percent  of  the  corresponding  aggregate  face  value)  but  barely  1  per- 
cent on  Study  P3,  a  difference  that  may  have  been  due  to  more  accurate 
reporting  on  the  latter  study  and/or  to  the  larger  proportion  of  policies 
not  validated. 

The  policies  not  reported  on  Study  P2  tended  to  be  considerably 
smaller,  on  the  average,  than  policies  that  were  reported,  with  the  result 
that  the  aggregate  face  value  of  nonreported  policies  is  an  even  smaller 
proportion  of  the  total,  less  than  8  percent.  This  more  favorable  experi- 
ence than  that  obtained  with  Study  PI  is  undoubtedly  due  in  part  to  the 
smaller  average  size  of  policies  encountered  in  the  later  study  since,  as  will 
be  noted  shortly,  nonreporting  rose  appreciably  with  the  size  of  the  policy. 
In  addition,  there  is  the  further  possibility  that  the  interviewing  staff  in 
Study  P2  was  better  than  that  in  Study  PI,  since  selection  procedures  and 
training  undoubtedly  benefited  from  the  earlier  study,  and  this  was  prob- 
ably even  more  true  with  Study  P3.  Indicative,  perhaps,  of  this  tendency, 
and  of  interest  in  its  own  right,  is  that,  of  the  policies  that  were  reported 
(and  validated),  figures  were  refused  on  37  in  Study  PI,  7  in  P2,  and  only 
5  in  P3. 

Where  face  value  was  reported  and  comparisons  could  be  made,  exact 
correspondence  occurred  on  almost  two- thirds  of  the  policies  on  PI,  three- 
fourths  of  the  policies  on  P2,  and  nearly  nine-tenths  of  those  on  P3.  On 
the  other  hand,  one-fifth  of  the  reports  in  Study  PI  deviated  more  than  50 
percent  from  the  true  figure.  However,  on  the  average  the  largest  errors 
were  not  associated  with  the  largest  policies.  Those  with  fully  correct 
reported  face  value  were  much  smaller  than  the  others,  but  the  largest 
policies  clearly  were  more  likely  to  be  characterized  by  moderate  errors 
in  face  value  than  by  large  errors,  something  to  which  we  will  return 
shortly. 

Response  errors  generally  averaged  lower  in  Studies  P2  and  P3  than 
in  Study  PI  and  also  occurred  less  frequently.   However,  large  errors  were 
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relatively  more  substantial  when  they  did  occur  in  the  later  studies,  par- 
ticularly errors  of  more  than  50  percent.  On  balance,  such  errors  pro- 
duced overestimates  of  233  percent  in  Study  P3  and  of  90  percent  in  Study 
P2,  as  compared  with  overestimates  of  50  percent  in  Study  PL 

It  is  also  interesting  to  note  that  the  response  errors  tended  to  be  pri- 
marily in  the  direction  of  overstatement,  although  progressively  less  so 
on  the  later  studies.  From  an  aggregative  point  of  view,  these  overstate- 
ments help  offset,  to  some  extent,  the  face  value  of  policies  not  reported, 
as  is  clear  from  the  following  comparison  for  Study  PI  (estimating  the 
"reported"  figure  for  face  value  refusals  on  the  basis  of  reported  face 

value  policies  for  which  this  figure  was  given)  : 

Percentage  of 

Reported             Actual  discrepancy 

Face  value  of  policies  reported $1,423,348  $1,254,342  -13.5 

Policies  reported,  face  value  refused 226,033           300 ,  562  24.8 

Face  value  of  policies  not  reported 582,121  100.0 

Total $1,649,381  $2,147,025  23.2 

In  other  words,  an  aggregative  estimate  of  face  value  based  on  these 
two  sets  of  data  would  have  resulted  in  an  underestimate  of  23  percent, 
mainly  due  to  nonreporting  of  insurance.  To  be  sure,  the  average  face 
values  of  policies  not  reported  and  of  face  value  refusals  are  substantially 
higher  than  the  average  face  value  of  the  policies  that  were  reported.  For 
this  reason,  we  find  that  the  estimated  average  face  value  of  validated 
life  insurance  policies  understates  the  true  figure  by  only  6  percent.93 

In  the  case  of  Study  P2,  the  interview  and  the  institutional  aggregates 
correspond  very  closely,  as  shown  by  the  following  data : 

Percentage  of 
Reported  Actual        discrepancy 

Face  value  of  policies  reported $     985,134         $916,210       -7.5 

Policies  reported,  face  value  refused 27,363  8,480   —222.1 

Face  value  of  policies  not  reported 75,017       100.0 

Total $1,012,497         $999,707       -1.3 

The  1  percent  overstatement  of  the  aggregate  is  seen  to  reflect  the 
(offsetting)  overstatement  both  of  face  value  of  policies  reported  and  of 
face  values  imputed  when  this  figure  was  refused,  offset  against  the  rela- 
tively small  face  value  of  policies  not  reported. 

For  Study  P3,  the  correspondence  is  even  closer,  and  for  the  same 
reasons  as  before : 

03  The  difference  between  this  figure  and  the  23  percent  understatement  of 
the  aggregate  is  primarily  due  to  the  fact  that  the  base  number  of  policies  for  the 
average  reported  size  of  policy  is  so  much  less  than  for  the  average  actual  size 
(233  versus  326). 
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Percentage  of 
Reported  Actual        discrepancy 

Face  value  of  policies  reported $778,395  $744,977  -4.5 

Policies  reported,  face  value  refused 14,575  9,500  —53.4 

Face  value  of  policies  not  reported 44,200  100.0 

Total $792,970  $798,677  0.7 

Allocating  the  sources  of  error  among  the  three  possible  causes  (non- 
response  bias  is  unknown)   yields  the  following  percentage  distributions 

for  the  three  studies: 

Study  PI         Study  P2         Study  P3 

Error  due  to  nonreporters 119.4  —591.8  774.5 

Inaccurate  estimates  of  those  refusing 

balances 15.3  149.0  -88.9 

Response  error -34.7  542.8  -585.6 

Total 100.0  100.0  100.0 

In  view  of  the  earlier  material,  it  is  not  surprising  to  find  nonreporting 
to  be  the  principal  cause  of  understatement,  and  face  value  refusals  and 
response  errors  to  be  offsetting  causes  of  overstatement.  The  large  size 
of  the  percentage  errors  in  the  two  later  studies  derives  from  the  small 
relative  error  in  the  sample  estimate  of  the  aggregate.  Entering  into  the 
denominator,  this  figure  serves  to  inflate  sharply  the  error  ratios. 

The  effect  of  these  various  types  of  errors  on  the  size  distribution  of 
policies  is  shown  in  Table  37.  For  Study  PI,  this  table  indicates  that  a 
much  higher  proportion  of  policies  that  were  not  reported  —  although  not 
of  policies  for  which  the  face  value  was  refused  —  tended  to  be  of  large 
size.  Thus,  5  percent  of  policies  that  were  reported  had  a  face  value  of 
more  than  $22,500,  compared  with  20  percent  of  policies  that  were  not 
reported  and  3  percent  of  policies  reported  but  for  which  the  face  value 
was  refused. 

In  the  case  of  face  value  of  policies  that  were  reported,  errors  in  the 
distributions  appear  to  be  relatively  slight.  There  is  some  tendency  for 
policies  that  were  overstated  to  be  more  frequently  above  $22,500  in  face 
value  and  for  policies  that  were  understated  to  be  more  frequently  in  the 
lower  range,  under  $2,500  in  face  value.  However,  neither  difference  is 
particularly  substantial,  with  the  result  that  the  reported  and  actual  dis- 
tributions by  face  value  of  those  policies  that  were  reported  coincide 
closely  (columns  6  versus  9).  Even  more  significant  is  the  close  corre- 
spondence between  the  distribution  of  reported  face  values  (column  2) 
and  that  of  all  policies  (column  9),  the  latter  including  policies  not  re- 
ported. Clearly,  in  this  distributional  sense,  response  (including  non- 
reporting)  errors  are  of  little  consequence. 

For  Studies  P2  and  P3,  Table  37  shows  that,  contrary  to  the  finding 
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for  Study  PI,  the  policies  that  were  not  reported,  or  for  which  the  face 
value  was  refused,  tended  to  be  very  much  smaller  than  those  that  were 
reported.  Thus,  roughly  one-fourth  of  the  P2  policies  in  the  former  group 
had  a  face  value  of  less  than  $1,000,  which  was  true  for  only  10  percent 
of  the  policies  that  were  reported. 

Despite  these  differences,  the  size  distribution  of  policies  based  on  the 
respondent  reports  on  these  two  studies  (column  2)  is  seen  to  correspond 
very  closely  with  the  true  size  distribution  of  all  policies  (column  9),  in- 
cluding those  that  were  not  reported.  The  only  difference  of  any  mag- 
nitude —  and  this  tends  to  be  small  —  is  a  tendency  to  understate  the 
proportion  of  policies  having  a  face  value  of  less  than  $1,000. 

The  moderate  nature  of  these  errors  in  the  frequency  distribution  com- 
parisons is  unfortunately  not  fully  reflected  in  the  estimates  of  the  true 
standard  error  of  the  mean.  Carried  out  in  the  manner  discussed  in  pre- 
vious chapters,  the  computations  produce  the  following  results: 

Study  PI     Study  P2     Study  P3 

k 1.20  .55  .70 

r -.02  .07  -.08 

Apparent  standard  error  of  the  mean  (ay)      $601  $503  $180 

True  standard  error  of  the  mean  (\S~MSE)     $942  $573  $219 
Response-error-free  standard  error  of  the 

mean  (<ri) $487  $476  $172 

In  Study  PI,  the  high  value  of  the  bias  relative  to  the  apparent  stand- 
ard error  serves  to  bring  about  an  underestimate  of  the  true  standard 
error  of  more  than  one-third.  Furthermore,  if  the  response  errors  could 
have  been  eliminated,  the  true  standard  error  would  have  been  reduced 
to  almost  half  its  present  size.  In  Study  P2,  the  relatively  small  errors  in 
the  aggregates  nevertheless  are  sufficient  to  produce  a  14  percent  dis- 
crepancy between  apparent  reliability  and  true  reliability.  The  discrepancy 
is  due  only  in  part  to  the  bias,  since  the  low  (positive)  correlation  between 
the  response  errors  and  the  true  values  accounts  for  roughly  one-third  of 
the  discrepancy. 

In  Study  P3,  the  apparent  standard  error  of  the  average  face  value, 
$180,  represents  an  understatement  of  the  true  standard  error  of  these 
data  of  18  percent.  Furthermore,  although  the  correlation  between  the 
response  errors  and  the  true  values  is  very  low,  elimination  of  the  response 
errors  would  have  reduced  the  true  standard  error  by  21  percent,  as  a 
result  of  the  moderately  high  variance  of  these  errors. 

The  values  of  k  in  these  studies  suggest  that  the  bias  in  the  usual 
manner  of  constructing  the  95  percent  confidence  interval  is  of  moderate 
proportions.  This  bias  varies  from  a  relatively  low  value  for  Study  P2, 
for  which  the  usual  95  percent  confidence  interval  has  a  probability  of 
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containing  the  true  parameter  of  .91,  to  Study  PI,  where  this  probability 
falls  to  .76.  Thus,  bias  is  present,  but  to  nowhere  near  the  extent  ob- 
served for  other  assets  and  debts. 

Determinants  of  Response  Error 

Two  series  of  tests  were  carried  out  to  measure  the  extent  of  associ- 
ation of  different  respondent  and  interview  characteristics  with  response 
error.  One  series  of  tests  sought  to  determine  which  factors  were  associ- 
ated significantly  with  a  tendency  not  to  report  particular  policies.  A 
dichotomy  was  employed  for  this  purpose,  namely,  "policies  reported" 
and  "policies  not  reported."94 

The  second  series  of  tests  sought  to  measure  the  characteristics  associ- 
ated with  different  degrees  of  response  error.  Only  policies  for  which  the 
face  value  was  reported  by  both  the  respondent  and  the  institution  were 
included  in  this  test.  The  test  itself  was  carried  out  on  a  savings-unit  basis, 
combining  all  such  policies  for  each  respondent  and  basing  the  response 
error  figure  on  the  percentage  difference  between  the  actual  cumulated 
face  value  of  these  policies  and  the  reported  face  value. 

The  results  obtained  with  the  first  series  of  tests  designed  to  measure 
factors  influencing  nonreporting,  are  shown  in  Table  38. 95  The  charac- 
teristics tested  are  classified  under  four  general  headings,  pertaining  to 
characteristics  of  the  savings  unit,  cooperativeness  in  the  interview  (sub- 
jective as  well  as  objective  ratings),  the  circumstances  of  the  interview, 
and  the  experimental  variables. 

As  is  evident  from  this  table,  various  characteristics  under  each  of  these 
four  headings  are  found  to  be  related  significantly  to  the  reporting  of 
policies.  Of  the  savings  unit  characteristics  tested  in  Study  PI,  significant 
factors  are  seen  to  be  education,  age,  and  size  of  policy.  Perhaps  even 
more  interesting  is  the  fact  that  the  nature  of  these  relationships  is  not 
always  what  might  be  expected.  Thus,  the  least-accurate  groups  of  re- 
spondents were  those  with  a  high  school  education  and  those  who  were 
either  relatively  young  (under  35)  or  much  older  (over  65) .  Furthermore, 
policies  not  reported  tended  to  fall  most  frequently  either  in  the  largest 
size  category  or  in  the  smallest  size  category.  These  results  (as  well  as 
further  tabulations)  appear  to  suggest  that  two  diverse  groups  are  more 
guilty  than  others  of  nonreporting  —  those  who  are  relatively  young  and 
have  fairly  small  policies  and  those  who  are  much  older,  have  a  high 
school  education,  and  own  relatively  large  policies. 

94  If  a  respondent  reported  some  policies  but  not  others,  he  was  classified  as  a 
nonreporter. 

95  No  attempt  was  made   to  explore  factors  associated  with  nonreporting  in 
Study  P3  because  of  the  very  small  number  of  policies  (15)   in  this  category. 


Table  38.   Association  Between  Selected  Characteristics  and 
Reporting  of  Insurance,  Studies  PI  and  P2 


Characteristic 


Savings  unit 

characteristics 

Education 

Age 

Size  of  savings  unit .... 

Occupation 

Income 

Home  ownership 

Value  of  home 

(if  owned) 

Nationality 

Marital  status 

Size  of  policy 

Size  of  savings  unit .  .  .  . 
Cooperativeness 

Accuracy  of  information 
Completeness  of 

information 

Use  of  records 

Remained  in  panel 

after  Wave  1 

Interest  in  study 

Items  refused 

Circumstances  of 
interview 
Location  of  interview 
(home  versus  other) .  . 


Length  of  interview . 


Number  of  contacts 
before  interview.  . 


Status  of  respondent  in 

savings  unit 

Experimental  variables 

Type  of  advance  letter .  . 

Structured  versus  un- 
structured approach. . 


Dollar  amounts  versus 
no  dollars 

Assets  first  versus 
debts  first 

Holdings  first  versus 
change  first 

Forms  in  interview 
versus  no  forms. . .  . 

All  assets  versus 

some  assets 


Study  PI 


Study  P2 


Chi- 
square 


8.57 
19.14 


2.85 
3.10 
2.52 
1.44a 

3.32 
1.19 
l.lla 
8.91 


.09a 

2.29a 
1.24a 

1.03a 


3.02a 
17.28 

23.88 

1.65 

1.36a 

5.48a 

.42a 
.48a 


Degrees 

of 
freedom 


.63 


Level  of 
signifi- 
cance 


.03 
Less 
than 
.01 
.73 
.44 
.64 
.15 

.35 
.76 
.27 
.03 


.94 

.02 
.21 

.30 


Less 
than 

.01 
Less 
than 

.01 

Less 
than 
.01 

.45 

.17 

Less 

than 

.01 

.67 


Chi- 
square 


2.55 
6.41 


4.52 
4.77 
1.34a 


1.78a 
2.04 

.75 

1.22a 
.61a 


.06a 
.23a 


1.13a 
.47 

3.20 
.09a 


2.60a 

1.23a 
1.12 


Degrees 

of 
freedom 


Level  of 
signifi- 
cance 


.47 
.09 


.35 
.31 
.18 


.08 
.37 

.70 

.22 
.54 


.95 
.82 


.26 
.79 

.20 
.93 


Less 

than 

.01 

.22 

.27 


a  The  t  test  rather  than  the  chi-square  test  was  used  in  this  case. 
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Except  for  size  of  policy,  it  is  interesting  to  note  that  none  of  the  usual 
indicators  of  affluence  appears  to  be  related  to  nonreporting.  This  in- 
cludes occupation,  income,  and  value  of  home.  Noteworthy  is  the  fact 
that  three  of  the  four  characteristics  having  to  do  with  circumstance  of 
the  interview  were  related  significantly  to  the  reporting  of  these  policies. 
As  one  might  expect,  the  shorter  interviews  tended  to  produce  a  higher 
frequency  of  nonreporting,  and  interviews  made  after  several  contacts  ex- 
hibited a  similar  characteristic.  Somewhat  unexpectedly,  however,  non- 
reporting  was  much  more  extensive  among  interviews  conducted  in  the 
respondent's  home  than  among  interviews  conducted  elsewhere,  usually 
in  the  respondent's  office.96 

Only  one  of  the  four  experimental  variables  appears  to  differentiate 
significantly  between  reporting  and  nonreporting  of  policies.  A  much 
lower  incidence  of  nonreporting  was  experienced  by  interviewers  using 
the  unstructured  approach.  This  coincides  with  the  reports  of  the  inter- 
viewers, many  of  whom  asserted  that  they,  as  well  as  the  respondents,  were 
more  at  ease  in  the  unstructured  situation.97 

Table  38  also  indicates  that  many  fewer  characteristics  were  signifi- 
cantly related  (at  the  .05  level)  to  nonreporting  of  life  insurance  in  Study 
P2  than  in  Study  PI.  Indeed,  none  of  the  cooperativeness  or  savings  unit 
characteristic  variables  in  Study  P2  are  significant,  not  even  size  of  policy. 
The  one  variable  to  be  almost  significant,  age,  is  in  this  position  because 
virtually  no  nonreporting  was  noted  among  those  65  years  of  age  or  more, 
quite  the  reverse  of  the  situation  in  Study  PI. 

Among  the  experimental  variables  and  those  reflecting  the  circum- 
stances of  the  interview,  a  similar  situation  is  apparent  with  only  one 
variable  —  type  of  form  used  —  being  statistically  significant.  Further- 
more, the  direction  of  the  relationship  contradicts  all  previous  findings: 
nonreporting  is  much  more  frequent  with  the  holdings  form  than  with 
the  change  form.  The  secondary  nature  of  the  present  validation  renders 
this  inference  particularly  suspect;  if  complete  nonreporting  of  holdings 
were  more  frequent  on  the  change  form  than  on  the  holdings  form,  as  is 
true  of  other  assets  and  debts,  the  secondary  validation  process  would 
ascribe  spuriously  high  accuracy  to  the  change  form. 

The  extent  of  association  between  some  of  the  respondent  and  inter- 
view characteristics  and  the  magnitude  of  response  error  is  summarized  in 

96  Of  course,  when  the  interview  is  conducted  in  the  office,  one's  wife  is  not 
present.   .   .   . 

97  In  evaluating  these  results,  the  secondary  nature  of  this  validation  should 
be  stressed.  For  this  reason,  these  tests  do  not  reflect  the  full  extent  of  nonreport- 
ing, and  it  is  not  clear  whether  or  not  undetected  nonreporting  might  vary  in  dif- 
ferent ways  from  the  ways  in  which  the  nonreporting  that  was  detected  was  found 
to  vary. 
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Table  39.  Not  all  the  characteristics  appearing  in  Table  38  were  used  for 
this  test,  because  inspection  of  that  table,  supplemented  with  cursory  ex- 
amination, indicated  virtually  no  chance  of  obtaining  significant  results 
with  such  characteristics  as  nationality  and  status  of  respondent  in  the 
savings  unit.  Three  categories  of  accuracy  were  used  in  this  test:  "over- 
stated by  10  percent  or  more,"  "within  10  percent,"  and  "understated  by 
10  percent  or  more." 

Table  39  shows  seven  variables  to  have  some  value  in  Study  PI  in 
discriminating  between  accurate  and  inaccurate  reports  at  or  near  the 
.05  probability  level.  Three  of  these  variables  are  savings  unit  character- 
istics, and  the  other  four  relate  to  cooperativeness  or  to  circumstances  of 
the  interview;  none  of  the  experimental  variables  is  significant. 

The  nature  of  these  relationships  indicates  that  accurate  reports  were 
more  frequently  given  by  persons  in  the  upper-income  levels,  those  with 
higher  education,  and  those  not  self-employed.  Reports  from  those  in 
low-income  levels  or  the  self-employed  were  much  more  likely  to  under- 
state the  true  face  value  than  reports  from  other  groups.  In  addition, 
accurate  reports  occurred  much  more  frequently  when  records  were  con- 
sulted, when  the  interview  took  place  at  the  respondent's  office,  when  the 
interview  was  relatively  long  (two  hours  or  more),  and  when  the  respon- 
dent was  rated  as  being  fairly  complete  in  his  report.  Here  again,  the  data 
highlight  the  somewhat  paradoxical  finding  that  the  interviewers  were 
much  more  likely  to  gauge  accuracy  correctly  when  asked  to  judge  the 
completeness  of  the  information  than  when  asked  to  judge  the  accuracy 
of  the  information. 

Even  fewer  variables  are  statistically  significant  on  Studies  P2  and  P3, 
but  those  that  are  significant  appear  to  be  meaningful.  Thus,  the  signifi- 
cance and  near-significance  of  education,  age,  and  income  in  Study  P2 
are  due  to  the  much  greater  tendency  toward  understatement  of  those 
with  less  education,  in  laboring  occupations,  and  in  a  middle-income  level. 

Use  of  records  is  significant  in  Study  P2  because  nonusers  were  much 
more  likely  than  users  to  understate  face  value.  On  the  other  hand,  the 
near-significance  of  accuracy  of  information  is  the  result  of  the  much 
higher  tendency  of  respondents  rated  "poor"  in  this  respect  to  overstate 
face  value.98 

In  Study  P3,  only  four  variables  were  found  to  be  related  significantly 
to  accuracy  of  reporting  if  the  .05  probability  level  is  used,  and  the  num- 
ber rises  to  five  if  the  level  of  significance  is  relaxed  to  .10.    Of  these 

98  It  is  interesting  to  note  that  in  this  particular  case  accuracy  is  almost  signifi- 
cant whereas  completeness  is  not.  However,  it  should  be  noted  that  the  "com- 
pleteness" rating  applies  to  savings  accounts  and  not  to  life  insurance;  such  a 
rating  was  not  obtained  for  the  latter. 
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variables,  three  relate  to  savings  unit  characteristics :  smaller  savings  units 
as  well  as  those  headed  by  the  less-well-educated  tended  to  understate 
face  value  more  frequently,  whereas  upper-income  savings  units  tended 
to  overstate  face  value  more  frequently.  In  the  case  of  education  and  in- 
come, these  findings  coincide  with  those  of  the  earlier  studies. 

The  one  variable  significantly  related  to  accuracy  of  report  among  the 

Table  39.   Association  Between  Selected  Characteristics  and 
Accuracy  of  Validated  Policies,  Studies  PI  to  P3 


Characteristic 


Chi-square 


Degrees  of 
freedom 


Level  of 
significance 


Study  PI 


Savings  unit  characteristics 

Education 

Age 

Income 

Size  of  savings  unit 

Size  of  policy 

Occupation 

Cooperativeness 

Accuracy  of  information 

Completeness  of  information. .  .  . 

Use  of  records 

Circumstances  of  interview 

Location  of  interview 

(home  versus  other) 

Length  of  interview 

Number  of  contacts 

Experimental  variables 

Type  of  advance  letter 

Structured  versus  unstructured. . 

Dollar  amounts  versus  no  dollars 


4 

.02 

4 

.54 

4 

Less  than 

.01 

4 

.34 

4 

.91 

4 

.08 

4 

.76 

4 

.03 

9 

Less  than 

.01 

Less  than 

.01 

4 

Less  than 

.01 

6 

.65 

2 

.31 

2 

.31 

2 

.21 

Study  P2 


Savings  unit  characteristics 

Education 

Age 

Occupation 

Income. .  .  .  ." 

Size  of  savings  unit 

Size  of  policy 

Home  ownership 

Cooperativeness 

Accuracy  of  information .  .  . 

Completeness  of  information 
(savings  accounts) 

Interest  in  study 

Use  of  records 

Items  refused 

Circumstances  of  interview 

Location  of  interview 

Length  of  interview 

Number  of  contacts 


2 

4 

.08 

.17 

4 

.12 

4 
4 

Less  than 
.83 

.01 

2 

1.00 
.54 

4 

.12 

2 

.81 

2 

.84 

2 

Less  than 

.01 

Less  than 

.01 

.65 

2 

.25 

2 

.58 
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Table  39.  (concluded) 


Characteristic 

Chi-square 

Degrees  of 
freedom 

Level  of 
significance 

Experimental  variables 

.95 
2.18 
3.64 
1.70 

2 
2 
2 
2 

.63 

Holdings  first  versus  change  first 

Forms  in  interview  versus  no  forms 

.35 
.17 
.44 

Stud) 

'  P3 

Savings  unit  characteristics 

6.51 
2.24 

2 
2 

.04 

Age 

.33 

Income 

12.54 

2 

Less  than  .01 

Size  of  savings  unit 

4.89 

2 

.09 

.43 

2 

.81 

Farm  ownership 

.74 

2 

.69 

Cooperativeness 

Accuracy  of  reportingb 

1.37 

2 

.50 

Completeness  of  savings  account 

reporting 

1.01a 

.31 

1.18 

4.13a 

2 

.56 

Items  refused 

Less  than  .01 

Circumstances  of  interview 

Location  of  interview 

4.13a 

Less  than  .01 

.32a 

.75 

1.22a 

.22 

Experimental  variables 

1.27 

2 

.53 

Type  of  form  used  first 

2 

a  The  t  test  rather  than  the  chi-square  test  was  used  in  this  case. 
b  On  checking  accounts  and  life  insurance  combined. 


measures  of  cooperativeness  —  items  refused  —  reflects  a  somewhat  pecu- 
liar type  of  relationship.  Although  highly  significant,  the  data  suggest 
that  people  who  refuse  one  or  more  items  of  information  are  more  likely 
to  be  accurate  than  those  who  do  not  refuse  any  information.  To  be  sure, 
such  a  relationship  is  conceivable,  so  far  as  the  former  group  is  composed 
of  people  who  either  give  accurate  information  or  prefer  to  give  no  infor- 
mation at  all.  However,  this  result  is  at  variance  with  the  direction  of  the 
relationship  for  the  same  variable  noted  in  Study  P2,  although  it  must  be 
remembered  that  the  composition  of  the  sample  in  Study  P3  is  very  dif- 
ferent from  that  in  Study  P2. 

Among  variables  reflecting  the  circumstances  of  the  interview,  location 
is  the  only  one  to  be  statistically  significant.  As  was  found  in  some  of  the 
previous  analyses,  these  results  support  the  possibility  that  much  more 
accurate  data  are  obtained  when  the  respondent  is  interviewed  elsewhere 
than  at  home. 
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Demand  Deposits" 

Validation  of  demand  deposits  presents  two  difficulties  not  encoun- 
tered heretofore.  First,  because  of  their  equivalence  to  cash  and  because 
they  are  held  primarily  for  transactions  and  liquidity  purposes,  demand 
deposits  tend  to  be  highly  volatile.  For  many,  if  not  most,  consumers  the 
fluctuation  in  the  monthly  balance  over  the  period  of,  say,  a  month  may 
exceed  substantially  the  average  balance  in  the  account.  Second,  at  any 
particular  moment  the  balance  in  a  given  account  according  to  institu- 
tional records  may  or  may  not  coincide  with  the  balance  in  that  account 
according  to  the  consumer's  records:  the  higher  is  the  velocity  in  the 
account,  the  less  likely  are  the  two  balances  to  coincide.  This  is  the  well- 
known  "float  problem,"  brought  about  by  the  interval  that  elapses  be- 
tween the  time  a  check  is  written  and  the  time  that  it  is  presented  for 
redemption  at  the  issuing  institution. 

Whether  or  not  the  bias  due  to  check-float  is  serious,  its  possible 
presence  serves  to  confuse  interpretation  of  such  validation  studies,  for 
segregation  of  errors  due  to  these  phenomena  from  nonsampling  errors  of 
the  operation  is  hardly  feasible.  Nevertheless,  as  far  as  is  known,  no  val- 
idation studies  of  demand  deposits  have  been  published,  and  it  was  there- 
fore considered  desirable  to  make  some  preliminary  efforts  in  this  direction. 

Two  of  the  operations  of  the  Consumer  Savings  Project  incorporated 
validation  of  demand  deposits.  Study  P2  included  secondary  validation 
of  these  holdings,  and  such  holdings  served  as  a  primary  validation  source 
for  part  of  the  data  obtained  in  Study  P3. 

The  Data 

Study  P2 

Toward  the  end  of  the  field  work  on  this  study,  an  attempt  was  made 
to  validate  checking  account  balances  reported  in  two  cooperating  insti- 
tutions. No  attempt  was  made  to  pick  up  unreported  accounts  of  panel 
members  not  mentioning  ownership  of  such  accounts.100  Hence,  this  anal- 
ysis relates  solely  to  response  error  in  balances  that  were  reported. 

One  institution  was  able  to  carry  out  this  check  only  partially,  with 
the  result  that  125  reported  balances  were  validated.  Lack  of  institutional 
reports  and  absence  of  complete  information  reduces  this  number  to  65, 
as  shown  by  the  following  breakdown : 

90  This  section  is  based  largely  on  the  author's  article,  "The  Reliability  of 
Consumer  Savings  of  Financial  Holdings:  Demand  Deposits,"  Journal  of  the 
American  Statistical  Association,  Vol.  61,  No.  313   (March,  1966),  pp.  91-103. 

100  This  was  not  feasible.  The  possibility  that  little  additional  information 
would  have  been  obtained  is  suggested  by  the  ability  of  the  institutions  to  locate 
only  one  additional  checking  account  for  the  panel  members  whose  balances  were 
validated. 
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Number 

Balance  not  reported  by  institution 48 

Balance  not  reported  by  respondent 10 

Report  not  comparable 2 

Report  comparable,  balance  obtained  from  both  sources 65 

Comparison  of  the  accounts  not  validated  by  the  institutions  with 
those  that  were  suggests  that  the  former  group  had  larger  accounts,  with 
balances  averaging  $1,893  compared  with  $992  for  the  validated  accounts 
(based  on  the  respondent  reports).  More  significant,  however,  may  be 
the  fact  that  the  nonvalidated  accounts  were  more  concentrated  at  both 
ends  of  the  (size)  distribution.  About  20  percent  of  the  nonvalidated  ac- 
counts had  balances  of  less  than  $100  compared  with  14  percent  of  those 
that  were  validated;  at  the  other  extreme,  the  relative  frequencies  of  ac- 
counts with  balances  of  $2,000  or  more  were  1 7  percent  and  1 1  percent, 
respectively.  Hence,  for  some  unknown  reason,  the  validated  balances  in 
this  instance  are  more  likely  to  represent  accounts  with  balances  closer  to 
the  average  than  accounts  with  very  large  or  very  small  balances. 

Study  P3 

Demand  deposit  balances  served  as  a  primary  validation  source  for 
part  (roughly  one-fifth)  of  the  sample  members  in  this  farm  panel  oper- 
ation; the  other  primary  validation  sources,  it  will  be  recalled,  were  debt 
and  savings  accounts.  As  was  true  with  other  primary  validations  in  panel 
operations,  it  was  feasible  to  validate  these  balances  throughout  the  course 
of  the  operation  (the  results  of  the  temporal  comparisons  will  be  presented 
in  Chapter  VI ) . 

Response  and  Nonresponse  Errors 

The  analysis  in  the  following  sections  will  focus  on  four  aspects  of 
response  and  nonresponse  errors  in  these  reports  of  demand  deposits  — 
their  effect  on  estimates  of  the  average  size  of  the  validated  accounts,  their 
effect  on  frequency  distributions  by  size,  their  effect  on  estimates  of  the 
reliability  of  the  mean,  and  the  extent  to  which  the  observed  errors  in  the 
data  are  attributable  to  check-float.  Results  will  be  presented  for  both 
surveys  so  far  as  this  is  feasible.  Principal  emphasis  will  be  placed  on 
Study  P3,  however,  because  it  contains  the  most  complete  information  on 
the  accuracy  of  reports  of  this  asset. 

In  the  next  two  sections,  discrepancies  between  the  reported  figures 
and  those  obtained  from  the  institutions  will  be  treated  as  nonsampling 
errors,  with  the  institutional  figures  given  as  "actual"  and  the  respondent 
figures  as  the  "reported"  data.  In  practice,  these  discrepancies  may  not 
reflect  the  true  extent  of  nonsampling  errors  insofar  as  the  data  contain 
matching  errors  and  are  affected  by  check-float.    Matching  errors  are 
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highly  unlikely  as  a  result  of  the  panel  nature  of  both  operations  and  the 
feasibility  of  checking  and  rechecking  to  ensure  matches;  as  was  noted 
previously,  two  reports  were  discarded  for  this  reason.  The  possibility  of 
check-float,  however,  cannot  be  ruled  out,  and  an  attempt  will  be  made 
to  measure  this  phenomenon. 

Effect  on  Estimated  Size  of  Balance 

Unfortunately,  data  on  nonreporting  are  available  only  from  Study 
P3.    These  data  are  presented  in  the  following  tabulation,  subdivided  by 

form  used : 

Holdings         Change 
form  form 

Respondents 33  36 

Reporters  (including  refusals  of  balance) ....  30  33 

Nonreporters 3  3 

Noncontacts 0  0 

Refusals  to  be  interviewed 6  2 

Total "39  38 

In  terms  of  the  number  of  accounts,  nonreporting  is  seen  to  amount 
to  roughly  10  percent  of  the  validated  accounts  of  those  interviewed. 
This  percentage  is  substantially  below  figures  which  varied  from  25  per- 
cent to  40  percent  nonreporting  for  such  assets  as  time  deposits  and  per- 
sonal debt  recorded  in  previous  studies.  Moreover,  nonreporting  was 
just  as  low  in  this  case  when  respondents  were  asked  about  changes  as 
it  was  when  they  were  asked  about  holdings;  this  is  contrary  to  the  situa- 
tion observed  for  other  assets,  for  which  there  was  a  tendency  for  non- 
reporting  to  be  much  higher  because  of  the  inclination  of  respondents  not 
to  report  holdings  with  little  or  no  change.  It  is  interesting  to  note,  how- 
ever, that  refusals  are  more  frequent  (though  not  significantly  so  at  the 
.05  percent  probability  level)  when  the  more  stringent  holdings  form  is 
used,  a  finding  in  line  with  past  experience. 

A  broader  picture  of  the  errors  in  the  balances  reported  by  both 
samples  is  shown  in  Table  40.  As  was  noted  previously,  the  data  for 
Study  P3,  based  on  primary  validation  techniques,  cover  all  forms  of 
these  errors,  whereas  the  data  for  the  urban  study,  obtained  by  secondary 
validation,  relate  only  to  discrepancies  in  balances  reported.  On  the  farm 
study,  the  reported  figure  for  the  total  sample  is  obtained  by  assigning  the 
same  average  balance  as  reported  by  respondents  who  gave  balances  to 
those  respondents  who  refused  to  cite  any  balances  or  to  grant  an  inter- 
view; there  were  no  noncontacts.  Other  allocation  techniques  could 
conceivably  have  yielded  a  higher  balance,  although  the  present  sample 
is  too  small  for  this  purpose.  Use  on  other  assets  of  such  techniques  as 
allocating  average  balances  after  classifying  the  sample  into  particular 
strata  suggests  that  they  would  have  had  relatively  little  effect. 
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Table  40.  Average  Size  of  Account,  by  Outcome 
of  Interview  Attempt 


Category 


Number 

of 
accounts 


Average  size  of 
account 


Actual      Reported 


Percentage 
of  deviation, 

reported 
from  actual8 


Study  P3 


Total  sample .  .  .  , 
Respondents.  .  .  . 

Nonreporters .  . 

Figure  refused. 

Figure  given.  . 
Nonrespondentsb . 


24 

$1,780 

$1,350 

22 

1,611 

1,350 

2 

266 

0 

1 

2,030 

19 

1,731 

1,484 

2 

3,642 

24.2 
16.2 
100.0 

14.3 


Study  P2 


Respondents.  .  .  . 
Figure  refused . 
Figure  given .  . 


75 

$1,195 

$  992 

10 

2,280 

65 

1,029 

992 

16.8 
3.6 


a  Actual-reported 

Actual 
b  Both  refusals;  there  were  no  noncontacts. 


On  the  more  comprehensive  farm  study,  the  average  balance  appears 
to  have  been  understated  by  roughly  25  percent.  This  discrepancy 
represents  partly  the  tendency  for  the  average  actual  balances  of  the 
nonrespondents  to  exceed  considerably  the  actual  average  balance  of  the 
respondents  and  partly  the  underestimation  by  respondents  of  their  hold- 
ings. In  the  latter  instance,  it  is  interesting  to  note  that  in  both  samples 
the  extent  of  underestimation  of  balances  by  respondents  is  roughly  one- 
sixth.  Moreover,  both  samples  also  indicate  that  balances  for  which  the 
figure  is  refused  tend  to  exceed  the  actual  balances  of  people  who  report 
these  figures,  substantially  so  in  the  urban  study  (in  the  farm  study  there 
is  only  one  such  refusal).  The  discrepancy  resulting  from  imputing  a 
balance  to  those  refusals  on  the  basis  of  the  balances  that  were  reported 
tends  to  be  increased  because  of  the  tendency  of  the  reporters  to  under- 
estimate their  own  balances,  although  this  underestimation  tends  to  be 
relatively  small. 

Numerical  estimates  of  the  relative  importance  of  these  different 
sources  of  error  result  in  the  following  error  allocation : 

Discrepancies  in  estimated  balances  of  nonrespondents 44.2% 

Error  due  to  nonreporters 5.1 

Inaccurate  estimates  of  those  refusing  balance 5.3 

Discrepancies  in  the  balances  reported  by  the  respondents 45 . 4 

Total 100.0% 
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Table  41.  Distribution  of  Balances  in  Validated  Accounts,  by  Size 

Study  P3 


Reported  size 

Actual  size 

Under 

$100 

$100 
to 

$299 

$300 

to 
$999 

$1,000 

to 
$2,499 

$2,500 
and  over 

Not  re- 
ported 

Total 

Under  $100 

$100  to  $299 

$300  to  $999 

$1,000  to  $2,499..  . 
$2,500  and  over .  .  . 

1 
4 
1 

3 

2 

2 

1 

5 

1 

i 

2 
1 

2 
4 
7 
4 
7 

Total 

0 

6 

5 

3 

5 

5 

24 

Study  P2  (reporters  only) 


Reported  size 

Actual  size 

Under 
$100 

$100 

to 
$299 

$300 

to 
$999 

$1,000 

to 
$1,499 

$1,500 

to 
$2,499 

$2,500 
and 
over 

Figure 
refused 

Total 

Under  $100 

$100  to  $299 

$300  to  $999 

$1,000  to  $1,499..  . 
$1,500  to  $2,499..  . 
$2,500  and  over .  .  . 

7 
2 
1 

2 
11 

1 

3 

16 

2 

5 
2 

3 
5 
1 

i 

3 

1 
3 
1 
2 
3 

10 
17 
20 
11 
10 
7 

Total 

10 

13 

22 

7 

9 

4 

10 

75 

These  results  support  previous  statements  that  the  primary  sources 
of  error  in  the  estimate  of  the  mean  were  the  disproportionately  large 
balances  of  the  nonrespondents  and  the  tendency  of  respondents  to  under- 
state balances  when  reported.  This  understatement,  although  relatively 
small,  is  magnified  in  the  present  case  by  the  proportionately  large  number 
of  sample  members  in  that  category. 

Effect  on  Frequency  Distributions  by  Size 

The  effect  of  these  various  discrepancies  on  frequency  distributions 
by  size  suggests  some  tendency  for  the  frequency  of  both  very  large  and 
very  small  accounts  to  be  understated  (Table  41),  for  much  the  same 
reasons  mentioned  earlier.  The  differences  are  particularly  pronounced 
at  the  upper  end  when  the  distribution  of  reported  balances  is  compared 
with  the  distribution  of  balances  for  the  total  sample,  including  refusals, 
nonreporters,  and  nonrespondents. 
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Table  42.  Response  and  Nonresponse  Errors  in  Reports 
of  Change  in  Demand  Deposit  Balances  on  Initial 
Interview,  Change  Form,  Study  P3 


Category 


Number 

of 
accounts 


Average  change 


Actual      Reported 


Percentage 

of 
discrepancy* 


Total  sample 

Respondents 

Nonreporters .  .  . 

Reporters 

Nonrespondents .  .  . 

a  Actual-reported 
Actual 


19 

19 

/ 

18 

0 


481 

4,013 

285 


■149 
■149 


-158 


131.0 
131.0 
100.0 
155.4 


The  accuracy  of  the  reports  of  change  in  demand  deposit  balances, 
for  which  data  are  available  from  the  other  subsample  on  the  farm  study, 
is  substantially  below  that  of  the  balances  themselves.  This  is  brought  out 
in  Table  42,  which  shows  data  for  change  comparable  with  the  data  on 
balances  in  Table  40.  The  data  in  Table  42  are  less  comprehensive  than 
those  in  the  former  case  because,  unfortunately  from  an  analytical  point  of 
view,  there  were  no  nonrespondents  in  the  subsample.  Nevertheless,  the 
results  are  clearly  not  favorable.  There  was  only  one  nonreporter  among 
the  19  respondents,  but  the  change  in  his  balance  was  the  largest  of  the 
group  —  an  increase  of  over  $4,000.  The  others  reported  an  average  de- 
crease in  their  balances  of  $158  in  the  preceding  three-month  period, 
whereas  their  average  balances  actually  increased  by  nearly  $300.  As  a 
result,  not  only  was  the  amount  of  change  substantially  understated,  but 
the  direction  of  change  was  missed  altogether. 

Further  investigation  reveals  that  the  poor  correspondence  of  the  aver- 
age changes  for  the  respondents  is  the  result  of  a  relatively  large  number 
of  substantial  errors  (Table  43).  Although  the  sample  sizes  are  small,  the 
discrepancies  are  striking.  First,  a  clear  tendency  is  apparent  for  respon- 
dents to  report  no  change  when  a  change  in  the  balance  did  indeed  occur. 
Second,  the  frequency  of  large  decreases  —  $500  or  more  - —  tends  to  be 
overstated.  Third,  the  frequency  of  large  increases  in  the  balances  tends 
to  be  understated  substantially.  Of  eight  deposit  balances  that  had  in- 
creased substantially  over  the  preceding  quarter,  only  three  were  reported 
as  increases- — one  other  was  not  reported,  two  were  reported  as  "no 
change,"  and  two  were  reported  as  having  decreased  by  $500  or  more. 

That  large  discrepancies  occurred  with  much  greater  frequency  in 
reports  of  change  than  in  reports  of  dollar  balances  is  apparent  from  the 
following  comparison  of  the  frequency  of  discrepancies  of  different  size: 
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Discrepancy  On  changes  On  balances 

Within  25  percent 2  8 

26-50  percent 2  4 

51-100  percent 8  4 

Over  100  percent 6  3 

18  19 

In  other  words,  only  about  one-tenth  of  the  change  reports  came 
within  25  percent  of  the  institutional  figure,  whereas  the  same  was  true  of 
almost  half  of  the  balance  reports.  More  than  three-fourths  of  the  change 
reports  differed  from  the  institutional  figure  by  50  percent  or  more,  and 
one- third  differed  by  100  percent  or  more.  In  contrast,  the  corresponding 
percentages  for  reports  of  balances  are  37  percent  and  16  percent, 
respectively. 


Table  43.   Distribution  by  Size  of  Actual  and  Reported  Changes 
in  Deposit  Balances,  Change  Form,  Study  P3 


Reported  change 

Not 
re- 
ported 

Actual  change 

Down 

No 
change 

Up 

Total 

$500 

or 
more 

$100 

to 
$499 

Less 
than 
$100 

Less 
than 
$100 

$100 
to 

$499 

$500 

or 
more 

Down  $500  or 
more 

Down  $100  to 
$499 

Down  less  than 
$100 

No  change .... 

Up  less  than 
$100 

Up  $100  to 
$499 

Up  $500  or     . 
more 

1 
1 
1 

2 

2 
1 

1 

1 
1 

1 
1 
2 

2 

1 

1 

1 

4 

3 
0 

1 
2 
8 

Total 

5 

4 

0 

6 

0 

2 

1 

1 

19 

Effect  on  Reliability 

Using  the  same  procedure  as  was  used  in  Chapter  IV  to  measure  the 
effect  of  these  nonsampling  errors  on  estimates  of  the  reliability  of  the 
mean,  the  following  estimates  are  obtained  for  Study  P3  : 

k 78 

r -.67 

Apparent  standard  error  of  the  mean  (ay) $317 

True  standard  error  of  the  mean  (\/MSE) $403 

Response-error-free  standard  error  of  the  mean  (a*) $413 
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Thus,  without  knowledge  of  the  validation  information,  the  standard 
error  of  the  mean  balance  for  this  sample  would  be  estimated  at  $317, 
whereas  the  true  standard  error  is  appreciably  more,  namely,  $403.  This 
true  standard  error  turns  out  to  be  much  the  same  as  what  would  be  ob- 
tained if  no  nonsampling  errors  were  present,  a  correspondence  brought 
about  because  of  the  tendency  for  the  increase  in  the  variance  due  to  the 
errors  to  be  offset  by  the  negative  correlation  between  the  actual  values 
and  the  errors.  (This  correlation  remains  strongly  negative  even  if  non- 
reporters  are  eliminated,  since  the  balances  of  the  nonreporters  were  very 
small;  see  Table  40.) 

The  value  of  k,  .78,  means  that  the  95  percent  confidence  interval 
applied  to  the  sample  observations  actually  had  a  probability  of  contain- 
ing the  true  parameter,  not  of  .93,  but  of  .87. 

Influence  of  Check-Float 

Ideally,  the  effect  of  check-float  would  be  investigated  by  obtaining 
information  on  the  balance  of  the  account  in  the  institutional  records 
and  the  balance  in  the  checkbook  of  the  respondent  showing  transactions 
not  yet  recorded  by  the  institution.  Any  difference  between  these  two 
balances  would  then  represent  check-float,  whereas  differences  between 
the  respondent  report  and  his  checkbook  balance  would  represent  response 
error.  However,  securing  such  data  was  out  of  the  question  and  a  more 
indirect,  less-exact  procedure  was  devised. 

The  basis  for  the  procedure  is  the  division  of  the  sample  into  two 
groups.  Group  I  consists  of  those  respondents  for  whom  any  observed 
discrepancies  are  clearly  attributable  solely  to  check-float  —  respondents 
who  consulted  records  and  gave  a  balance  to  the  nearest  dollar  or  nearest 
cent  and  for  whom  no  doubt  existed  regarding  the  accuracy  of  the  re- 
ported figures.  Group  II  consists  of  those  respondents  who  showed  reluc- 
tance to  cooperate  and  who,  in  most  instances,  rounded  off  figures  to  the 
nearest  $100  or  to  the  nearest  $1,000.  For  this  group  some  response  error 
is  highly  probable,  and  there  is  little  doubt  that  such  errors  will  be  com- 
pounded with  the  float  problem.  Included  among  this  group  are  people 
who  may  not  have  been  fully  cooperative  but  nevertheless  appeared  to  be 
making  some  attempt  to  provide  accurate  data.  Because  of  the  difficulty 
of  classifying  these  people  by  presence  of  response  error,  they  are  included 
in  this  second  category. 

The  approach  is  to  estimate  the  magnitude  of  the  float  on  the  basis 
of  the  Group  I  data.  The  resulting  estimate  is  then  compared  with  the 
discrepancy  from  Group  II,  and  any  overall  discrepancy  exceeding  that 
for  the  former  group  is  assumed  to  represent  response  error.  To  be  sure, 
there  is  little  evidence  that  the  two  groups  are  necessarily  comparable 
with  each  other,  a  basic  assumption  underlying  this  approach;  but  as 
will  be  shown  shortly,  some  allowance  can  be  made  for  differences. 
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Algebraically,  the  method  is  one  of  segmentation  of  variances.  If  ei 
represents  the  response  error  of  an  individual  in  the  ith.  group,  fi  the 
discrepancy  due  to  float,  and  d\  the  total  discrepancy,  we  have:101 

(5.1)  di  =  ei  +  fi. 

If  we  assume  absence  of  correlation  between  float  and  response  error,  it 
follows  that : 

(5.2)  ddf  =  <jef  +  07,2.102 

If  we  further  assume  that: 

e  =  zero  for  all  members  of  Group  I, 

e  differs  from  zero  for  some  members  of  Group  II,  and 

o-/,2  =  °h\ 

then  we  can  segregate  the  influence  of  float  and  that  of  response  error  on 
the  basis  that  o-/,2  =  a^2,  and  ae2  =  crd2  —  af2. 

However,  suppose  that  the  two  groups  are  not  comparable  because  the 
checking  account  transactions  of  one  group  differ  from  those  of  the  other. 
For  present  purposes,  the  most  pertinent  such  difference  is  likely  to  be  in 
the  average  balance  or  in  the  variability  of  the  average  balance,  since 
either  might  affect  the  magnitude  of  check-float.103  As  a  first  approxima- 
tion, one  supported  by  the  data,  let  us  assume  that  the  size  of  float  is 
related  to  the  average  balance,  e.g. : 

(5.3)  fi  =  a  +  {3  Xi, 
where  Xi  is  the  average  balance. 

It  then  follows  that  for  two  groups  with  different  average  balances, 

&x2 

(5-4)  <rf?  =  <rf*  —4-, 

and: 

<Te2  =  o-d2  -  o-/2. 

2  S  •'2 

Applying  this  method  to  the  data  leads  to  the  following  results: 


101  Strictly  speaking,  a  subscript  should  be  added  to  denote  each  individual  in 
the  group,  but  this  is  unnecessary  for  present  purposes. 

102  The  data  suggest  that  large  accounts  (over  $500)  tend  to  be  more  accurate 
than  small  accounts,  in  terms  of  percentage  error,  but  this  relationship  may  itself 
be  due  to  float  rather  than  to  response  error.  If  the  assumption  were  not  valid 
(2.2)  would  be  replaced  by: 

(5.2a)  o-rfj2  =  aef  4-  07. 2  4"  re.f.  ae.  ali 

In  that  case,  the  relative  influence  of  float  could  only  be  estimated  given  the  cor- 
relation between  float  and  response  error,  or  a  range  of  estimates  could  be  derived 
for  alternative  values  of  r. 

103  Pertinent  also  would  be  the  velocity  of  transactions,  but  no  data  were 
collected  on  this  subject. 
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Study  P2  Study  P3 

Group  I  Group  II  Group  I  Group  II 

di $           78  $           -18  $             217  $            256 

<7d2 36,805  1,175,136  92,789  1,161,312 

xi 873  1,162  1,060  1,970 

<7*2 724,118  4,899,640  1,399,439  4,378,333 

N 31  34  5  14 

Since  both  the  average  balance  and  the  variance  of  the  balance  are 
substantially  greater  for  the  possibly  inaccurate  group  (II)  than  for  the 
clearly  accurate  group  in  each  case,  Equations  (5.4)  are  applied  to  yield 

for  Study  P2:          c/22  =  249,036,  ae?  =  926,100,  and 
for  Study  P3:          o>82  =  290,303,  ae22  =  871,009. 

For  Study  P2,  this  leads  us  to  infer  that  21  percent  of  the  (variance 
in  the)  discrepancy  in  checking  account  balances  for  Group  II  is  attrib- 
utable to  check-float  and  79  percent  to  response  error.  Combining  the 
two  groups,  we  would  conclude  that  24  percent  of  the  total  variance  is 
due  to  check-float  and  76  percent  to  response  error.104  Thus,  it  appears 
in  this  case  that  response  error  was  of  much  greater  importance  than 
check-float  in  accounting  for  the  observed  discrepancies  in  checking 
bal 


ances. 


105 


For  the  farm  study,  the  data  suggest  that  25  percent  of  the  (variance 
in  the)  discrepancy  in  checking  account  balances  is  accounted  for  by 
check- float  and  75  percent  by  response  error.  Combining  the  two  sets  of 
data  yields  an  estimate  of  73  percent  as  the  proportion  of  total  variance 
in  the  observed  discrepancies  attributable  to  response  error  and  27  percent 
to  check-float.106  These  estimates  come  remarkably  close  to  the  76-24 
division  in  the  urban  study.107  Hence,  they  serve  to  reinforce  the  idea 
that  response  errors  may  be  a  much  more  important  explanation  of  dis- 
crepancies in  demand  deposit  balances  than  check-float. 

Farm  Assets 
Nature  of  Validation 

With   the   aid    of    information    obtained    from    cooperating    lending 

104  Allocation  of  response  error  to  the  total  variance  is  based  on  the  product 
of  the  proportion  of  Group  II  variance  due  to  response  error  (79  percent)  and  the 
contribution  of  the  Group  II  observations  to  the  total  sum  of  squares,  the  per- 
centage being  obtained  by  dividing  this  product  by  the  total   sum  of  squares. 

10a  To  judge  from  the  Group  I  data,  check-float  appears  to  have  been  of  minor 
importance  from  an  aggregative  point  of  view,  being  less  than  10  percent  of  the 
average  balance.  Even  individual  discrepancies  were  not  too  large,  the  average 
absolute  discrepancy  representing  13  percent  of  the  average  balance. 

106  The  combination  method  used  is  the  same  as  that  described  in  Footnote  104. 

107  In  this  case,  however,  d/x  for  Group  I  is  much  larger  than  before,  namely, 
about  20  percent.  Since  all  discrepancies  in  this  group  were  positive,  this  per- 
centage also  represents  the  average  magnitude  of  individual  discrepancies. 
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agencies,  a  variety  of  farm  assets  of  the  panel  members  of  Study  P3  were 
validated.  These  validations  were  carried  out  at  the  end  of  the  operation, 
focusing  on  information  obtained  on  Wave  5  interviews. 

From  a  conceptual  point  of  view,  this  procedure  was  tantamount  to 
primary  validation,  even  though  the  checks  were  made  on  an  ex  post 
basis.  The  reason  for  this  was  that  with  one  exception  ( stock  in  a  lending 
agency)  all  assets  were  physical  and  the  coverage  related  to  total  coverage 
of  each  type  of  asset.  For  the  latter  reason,  nonreporting  was  virtually 
nonexistent,  particularly  since  the  assets  served  in  many  instances  as  col- 
lateral for  the  loans.  Indeed,  one  would  expect  to  encounter,  if  anything, 
overreporting  rather  than  underreporting  in  such  a  situation.  At  the 
same  time,  these  data  do  not  possess  the  other  virtues  of  primary  valida- 
tion. Thus,  they  cannot  be  construed  as  representing  a  random  selection 
of  farmers  owning  each  of  the  assets  being  validated,  since  the  validation 
pertained  only  to  patrons  of  a  particular  type  of  lending  institution.  In 
addition,  the  institutional  data  themselves  were  based  to  a  large  extent  on 
information  supplied  by  the  individual  farmer,  although  this  information 
was  supplemented  by  checks  by  field  representatives  of  the  agency. 

From  a  practical  point  of  view,  certain  difficulties  arose  in  this  valida- 
tion not  encountered  in  any  previous  analyses.  The  principal  such  diffi- 
culty derived  from  the  fact  that  the  data  at  the  agency  were  not  con- 
tinuous; they  related  to  selected  times  at  which  the  assets  of  the  farmer 
happened  to  be  brought  up  to  date,  and  only  rarely  did  these  dates  coin- 
cide with  the  dates  of  the  interviews.  Hence,  some  interpolative  procedure 
was  required  to  estimate  the  farmer's  holdings  of  an  asset  on  a  particular 
date.  For  the  more  volatile  assets,  such  as  crops  and  certain  types  of  live- 
stock, such  interpolation  became  highly  tenuous  if  the  date  of  the  agency's 
records  and  the  date  of  the  interview  differed  substantially;  and  in  some 
instances  the  most  prudent  course  appeared  to  be  not  to  make  the  attempt. 

A  second  practical  difficulty  was  that  the  files  of  the  agency  were  at 
times  incomplete.  Primary  attention  appeared  to  be  given  to  those  assets 
serving  as  collateral  for  loans  or  to  those  assets  constituting  the  major  part 
of  the  farmer's  equity.  Third,  the  data  serving  as  the  basis  for  these  vali- 
dations were  considerably  fewer  than  the  number  of  panel  members  bor- 
rowing from  these  institutions,  partly  because  not  all  of  these  assets  were 
owned  by  every  farmer  and  partly  because  the  records  of  one  of  the  lend- 
ing agencies  (specializing  in  long-term  loans)  were  not  suitable  for  this 
type  of  analysis. 

Fourth,  the  determination  of  the  true  figure  was  in  many  cases  a 
matter  of  judgment.  With  dollar  amounts  this  is  self-evident,  for  value 
of  livestock  or  of  machinery  (and,  to  a  lesser  extent,  of  crops)  can  not  be 
determined  absolutely  until  the  asset  is  sold.    Even  when  numbers  are 


Validation  Studies:   Other  Assets  151 

Table  44.    Correspondence  of  Validated  Farm  Assets,  Study  P3 


Asset 


Measure 


Number  of 
observations 


Average  percentage 
of  discrepancy" 


Arithmetic      Absolute 


Acreage  (owners) 

Acreage  (renters) 

Automobiles  and  trucks .  . 

Other  machinery 

Hogs 

Cows  and  steers 

Other  livestock 

Crops 

Stock  in  lending  agency.  . 

a  Institution-respondent 
Institution 


Number  of  acres 

Number  of  acres 

Value 

Value 

Value 

Value 

Value 

Value 

Value 


34 
70 
78 
76 
46 
57 
13 
76 
76 


-2 
-20 
-9 
-2 
-10 
1 
-8 
12 


20 

14 
42 
33 
65 
44 
44 
134 
71 


involved  —  as  in  the  case  of  livestock  or  crops  —  differences  of  opinion 
may  arise,  since  exact  counts  are  not  often  made.  For  this  reason  the 
analysis  is  couched  primarily  in  terms  of  "correspondence"  of  information 
rather  than  "accuracy,"  with  the  base  as  the  institutional  figure,  which 
may  be  expected  to  be  the  least  biased. 

Magnitude  of  Error 

The  nine  categories  of  farm  assets  validated  and  the  average  errors 
obtained  in  each  category  are  shown  in  Table  44.  For  comparative 
purposes,  absolute  averages  of  the  individual  percentage  errors  are  shown 
as  well  as  arithmetic  averages.  These  absolute  averages  indicate  that 
substantial  discrepancies  occurred  between  respondent  reports  and  the 
institutional  data.  However,  the  average  discrepancies  vary  sharply  by 
asset,  ranging  from  14  percent  for  the  farm  acreage  of  renters  to  134 
percent  for  the  value  of  farm  crops.  Even  if  the  average  error  on  the 
value  of  crops  is  attributed  largely  to  differences  in  coverage,108  it  is  clear 
that  appreciable  individual  differences  exist  between  the  two  sets  of  data 
(as  shown  in  further  detail  in  Table  45,  p.  154) . 

It  is  also  clear  that  the  two  estimates  of  physical  quantity  exhibit  a 
discrepancy  considerably  less  than  that  in  the  estimates  of  value.  This  dif- 

108  This  figure  may  indeed  reflect  lack  of  comparability  between  the  interview 
data  and  the  institutional  data  more  than  anything  else.  As  reported  in  the  insti- 
tutional records,  value  of  farm  crops  was  meant  to  exclude  stored  crops,  whereas 
in  the  interviews,  information  on  stored  crops  was  sought  as  well  as  information  on 
other  crops.  Although  an  attempt  was  made  to  eliminate  stored  crops  from  the 
interview  data,  it  was  not  always  clear  whether  this  attempt  was  fully  successful. 
Moreover,  in  some  instances  there  was  a  strong  basis  for  believing  that  stored  crops 
were  included  in  the  institutional  data  (themselves  obtained  from  the  farmers), 
but  this  could  not  be  checked. 
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ference  could  be  attributed  to  the  particular  assets  covered,  but  it  more 
likely  reflects  the  fact  that  a  value  estimate  in  the  present  instance  neces- 
sitates both  an  estimate  of  physical  quantity  and  an  estimate  of  value  per 
unit;  and  these  two  different  estimates  do  not  seem  to  cancel  each  other. 

Also  apparent  from  Table  44  is  the  fact  that  these  large  absolute  errors 
do  tend  largely  to  cancel  each  other  in  the  aggregate,  so  that  the  arith- 
metic averages  are  not  much  different  from  zero  in  most  instances. 
Indeed,  all  but  one  of  these  averages  are  within  12  percent  of  the  insti- 
tutional averages.  However,  it  is  somewhat  surprising  to  note  that  seven 
of  these  nine  averages  tend  to  be  higher  for  the  interview  data  than  for 
the  institutional  data.  Since  many  of  these  assets  served  as  collateral  for 
the  loans  from  the  institution,  the  debtor  might  have  been  expected  to 
attempt  to  bias  upward  his  valuation  of  his  holdings.  Clearly,  however,  no 
such  tendency  is  apparent.  If  anything,  the  data  suggest  that  the  farmers 
were  more  comprehensive  in  reporting  holdings  in  the  interviews  (in  part 
undoubtedly  due  to  the  more  detailed  nature  of  the  interview) .  This 
tendency  receives  further  support  from  the  following  tabulation,  which 
shows  the  number  of  times  farmers  failed  to  report  particular  holdings  to 
the  institutions  and  the  number  of  similar  situations  in  the  interviews : 

Not  reported  Not  reported 

to  institution  in  interview 

Acreage  (owners) 4  0 

Acreage  (renters) 1  0 

Autos  and  trucks 1  0 

Other  machinery 3  0 

Hogs 4  3 

Cows  and  steers 2  3 

Other  livestock 33  1 

Crops 3  8 

Stock  in  lending  agency 1  21 

With  the  exception  of  crops  and  stock  in  the  lending  agency,  omis- 
sions appear  to  be  considerably  more  frequent  in  the  data  reported  to  the 
institution.  The  more  frequent  omission  of  crops  in  the  interview  than 
in  reports  to  the  institution  may  well  reflect  the  lack  of  comparability 
which  was  mentioned  in  the  previous  footnote.  The  frequent  omission 
of  stock  in  the  lending  agency  could  be  due  to  memory.  In  order  to 
borrow  from  these  agencies,  farmers  were  required  to  purchase  stock  in 
them  at  the  time  of  receipt  of  their  initial  loan.  Such  stock  constituted 
a  nominal  sum,  did  not  have  to  be  sold  at  any  particular  time,  and  there- 
fore could  be  easily  overlooked. 

Distributions  of  the  validated  farm  assets  by  degree  of  correspondence 
between  the  two  sources  are  shown  in  Table  45,  together  with  data  on  the 
true  average  value   (or  acreage)    in  each  category.    These  distributions 
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highlight  the  considerable  dispersion  of  the  discrepancies  for  all  assets  but 
farm  acreage.  Discrepancies  of  50  percent  or  more  exist  in  roughly  40 
percent  of  the  valuations  of  hogs,  of  crops,  and  of  stock  in  the  lending 
agency.  At  the  same  time,  these  distributions  also  suggest  that,  with  the 
exception  of  stock  in  the  lending  agency,  the  frequency  of  errors  is  dis- 
tributed more  or  less  symmetrically.  Only  for  stock  in  the  lending  agency 
are  there  considerably  more  large  errors  on  one  side  (understatements) 
than  on  the  other. 

Nevertheless,  for  several  assets  the  data  suggest  that  the  holdings  of 
those  erring  in  the  direction  of  understatement  tend  to  be  considerably 
higher  than  the  corresponding  holdings  for  the  overstaters.  For  example, 
the  average  value  of  the  hogs  of  farmers  overstating  this  holding  by  50 
percent  or  more  was  $810,  whereas  the  average  value  of  hogs  of  farmers 
understating  this  holding  by  50  percent  or  more  was  $3,773.  Much  the 
same  phenomenon  is  apparent  for  acreage  of  both  owners  and  renters  and 
for  value  of  automobiles  and  trucks,  for  other  machinery,  for  cows  and 
steers,  and  for  crops.  In  other  words,  as  is  true  with  financial  assets,  there 
is  a  tendency  for  the  more  affluent  to  understate  and  the  less  affluent  to 
overstate. 

The  effects  of  these  errors  as  well  as  of  nonreporting  on  the  size  distri- 
butions of  these  holdings  is  shown  in  Table  46.  In  view  of  the  fact  that 
nonreporting  was  more  frequent  in  the  institutional  records  for  most  assets, 
there  is  even  less  reason  than  before  for  assuming  that  the  size  distribution 
as  obtained  from  the  institution  provides  the  more  accurate  estimates. 
Nevertheless,  it  is  of  interest  to  observe  the  extent  to  which  the  two  size 
distributions  correspond.  As  is  evident  from  this  table,  such  correspon- 
dence is  close  for  all  assets  for  which  nonreporting  in  one  source  or  the 
other  was  relatively  small.  This  is  true  for  five  of  the  nine  assets  covered. 
In  addition,  for  a  sixth  asset,  hogs,  nonreporting  occurred  with  more  or 
less  equal  frequency  in  both  sources  and  here,  too,  close  correspondence 
is  apparent. 

Substantial  differences  exist  among  the  size  distributions  for  the  other 
three  assets:  for  other  livestock  because  of  extensive  nonreporting  to  the 
institutions;  for  stock  in  the  lending  agency  and  for  crops  because  of 
relatively  higher  nonreporting  in  the  interviews. 

It  is  interesting  to  note,  however,  that  if  the  "zero"  category  were 
eliminated  from  these  distributions,  no  appreciable  differences  would  be 
apparent  for  any  of  the  nine  assets. 

Reliability  of  Error  Estimates 

Computations  of  the  extent  to  which  the  apparent  standard  error  of 
the  mean  coincides  with  the  true  standard  error  for  each  of  the  nine  farm 
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assets  yields  mixed  results.  It  is  evident  from  Table  47  that  the  estimates 
virtually  coincide  for  some  assets  (for  acreage  and  for  two  of  the  livestock 
categories)  but  understate  substantially  the  true  standard  error  for  such 
assets  as  automobiles  and  trucks,  hogs,  crops,  and  stock  in  the  lending 
agency.   For  the  latter  group  of  assets,  the  true  confidence  interval  differs 


Table  45.  Distribution  of  Validated  Farm  Assets, 
by  Correspondence  of  Responses,  Study  P3 


Acreage  (owners) 

Acreage  (renters) 

Degree  of  correspondence 

Percent- 
age of  ob- 
servations 

Actual 

average 

acres 

Percent- 
age of  ob- 
servations 

Actual 

average 

acres 

Over  by  50  percent  or  more 

Over  by  25  to  49  percent 

Over  by  10  to  24  percent 

Within  10  percent 

Under  by  10  to  24  percent 

Under  by  25  to  49  percent 

Under  by  50  percent  or  more 

3 

85 

3 
9 

80 

153 

146 
233 

6 
1 
6 
71 
4 
9 
3 

321 
225 
325 
287 
375 
383 
1,062 

Total 

100 

158 

100 

326 

Automobiles  and 
trucks 

Other  machinery 

Degree  of  correspondence 

Percent- 
age of  ob- 
servations 

Actual 

average 

value 

Percent- 
age of  ob- 
servations 

Actual 

average 

value 

Over  by  50  percent  or  more 

21 
6 

13 

32 
9 

12 
8 

$1,533 
1,956 
2,185 
2,613 
2,179 
2,173 
3,016 

12 
13 
13 
25 
20 
12 
5 

$  6,966 
7,639 

9,615 

10,101 

Under  by  10  to  24  percent 

11,846 
9,502 

Under  by  50  percent  or  more 

10,511 

Total : 

101 

$2,403 

100 

$  9,624 

Hogs 

Cows  and  steers 

Degree  of  correspondence 

Percent- 
age of  ob- 
servations 

Actual 

average 

value 

Percent- 
age of  ob- 
servations 

Actual 

average 

value 

Over  by  50  percent  or  more 

Over  by  24  to  49  percent 

20 
13 
37 
17 
11 
13 
20 

$     810 
1,414 
1,333 
3,299 
2,811 
1,249 
3,773 

14 
14 
14 
25 
12 
7 
14 

$3,642 
7,466 

5,261 

3,968 

Under  by  10  to  24  percent 

6,177 

Under  by  25  to  49  percent 

6,925 

Under  by  50  percent  or  more 

6,774 

Total 

101 

$2,210 

100 

$5,467 
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Table  45.  (concluded) 


Other  livestock 

Value  of  stock  in 
lending  agency 

Degree  of  correspondence 

Percent- 
age of  ob- 
servations 

Actual 

average 

value 

Percent- 
age of  ob- 
servations 

Actual 

average 

value 

Over  by  50  percent  or  more 

Over  by  25  to  49  percent 

Over  by  10  to  24  percent 

Within  10  percent 

Under  by  10  to  24  percent 

Under  by  25  to  49  percent 

Under  by  50  percent  or  more 

15 
23 

8 

8 

39 

8 

$     210 
152 

'    36 

312 

367 

50 

9 
1 
5 
22 
12 
15 
36 

$       471 
830 
490 
544 
1,131 
835 
685 

Total 

101 

$     239 

100 

$       676 

Crops 

Degree  of  correspondence 

Percent- 
age of  ob- 
servations 

Actual 

average 

value 

Over  by  50  percent  or  more 

Over  by  24  to  49  percent 

Over  by  10  to  24  percent 

Within  10  percent 

Under  by  10  to  24  percent 

Under  by  25  to  49  percent 

Under  by  50  percent  or  more 

18 

7 

8 

22 

8 

15 

22 

$1,667 
2,395 
3,098 
3,717 
3,219 
3,751 
4,904 

Total 

100 

$3,435 

substantially  from  the  .95  confidence  interval  that  would  be  expected  by 
using  the  range  of  the  mean  plus  and  minus  1.96  apparent  standard  errors. 

The  distortion  would  have  been  considerably  larger  were  it  not  for 
some  substantial  negative  correlations  between  the  true  values  and  the 
errors.  These  correlations  serve  to  offset  partially  the  increased  sampling 
variance  caused  by  discrepancies  between  the  institutional  and  the  indi- 
vidual reports.109 

Column  6  of  Table  47  shows  that  in  many  instances  appreciable  reduc- 
tions in  the  standard  error  of  the  mean  would  have  taken  place  had  there 
been  no  discrepancy  between  the  two  sets  of  data.  This  is  true  of  all  of 
the  assets  but  acreage  and  other  livestock.  It  is  interesting  to  note, 
however,  that  in  the  latter  case,  the  standard  error  of  the  mean  in  the 
absence  of  error  would  have  been  considerably  larger  than  when  response 

109  It  should  be  stressed  that  these  correlations  appear  to  be  due  primarily  to 
two  or  three  extreme  observations  in  each  case  and  do  not  seem  to  typify  all  the 
observations.  If  these  extreme  observations  were  removed,  the  correlations  would 
become  virtually  zero. 
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error  was  present,  because  of  the  high  negative  correlation  between  the 
true  values  and  the  response  errors. 

Comparative  Evaluation 

The  nature  of  the  validation  results  obtained  in  this  chapter  vary 
substantially  with  the  type  of  asset,  influenced  no  doubt  by  the  type  of 
validation.  Thus,  the  low  frequencies  of  nonreporting  on  life  insurance 
may  be  due  in  part  to  the  secondary  nature  of  the  validations.  The  same 
may  be  said  of  the  low  nonreporting  of  demand  deposit  accounts  in  Study 


Table  46.  Percentage  Distribution  of  Validated  Farm  Assets, 
by  Size  and  by  Source,  Study  P3 


Item 

Inter- 
view 

Institu- 
tion 

Item 

Inter- 
view 

Institu- 
tion 

Farm  acreage  (owners) 

Farm  acreage  (renters) 

0 

18 

13 

21 

26 

8 

5 

8 

11 
11 
13 
24 
18 
11 
8 
5 

0 

Under  150 

9 
25 
34 
13 
13 

7 

1 

Under  50  . . 

7 

50  to  74 

1 50  to  249 

28 

75  to  99 

100  to  199. 

250  to  349 

350  to  449 

35 
10 

200  to  299 

450  to  549 

550  and  over 

Total 

10 

300  to  399 

400  and  over 

9 

Total 

99 

101 

101 

100 

Value  of  automobiles  and  trucks 

Value  of  other  machinery 

$0.    . 

13 
19 
13 
10 
14 
13 
9 
10 

1 
15 
18 
18 
15 

9 
11 

4 

9 

$0 

1 

5 
35 
35 
19 

4 

4 

Under  $1,000 

$1,000  to  $1,499 
$1,500  to  $1,999 
$2,000  to  $2,499 
$2,500  to  $2,999 
$3,000  to  $3,999 
$4,000  to  $4,999 
$5,000  and  over 

Under  $1,000 

$1,000  to  $2,499 
$2,500  to  $7,499 

$7,500  to  $12,499 

$12,500  to  $22,499.  .  . 
$22,500  and  over .... 

Total 

3 

3 

30 

38 

19 

4 

Total 

101 

100 

99 

101 

Value  of  hogs 

Value  of  cattle 

$0. 

6 
22 
14 
20 
16 
20 

2 

8 
14 
20 
18 
18 
14 

8 

$0 

3 
10 
14 
31 
22 
14 

7 

3 

Under  $500 

Under  $1,000 

$1,000  to  $2,499 
$2,500  to  $4,999 
$5,000  to  $7,499 

$7,500  to  $12,499 

$12,500  and  over 

Total 

7 

$500  to  $999 

$1,000  to  $1,499 
$1,500  to  $2,499 
$2,500  to  $5,499 
$5,500  and  over 

17 
29 
22 
15 

7 

Total 

100 

100 

101 

100 
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Table  46.  (concluded) 


Item 


Inter- 
view 


Institu- 
tion 


Value  of  other  livestock 


Under  $50 

$50  to  $99 

$100  to  $299 

$300  to  $999 

$1,000  and  over. 

Total 


2 
28 
13 
37 
15 

4 


99 


72 
2 
7 

11 
7 
2 


101 


Item 


Inter- 
view 


Value  of  crops 


Under  $1,000 

$1,000  to  $1,499.  .  .. 
$1,500  to  $4,999.  .  .. 
$5,000  to  $9,999.  .  .  . 
$10,000  and  over.  .  . 

Total 


100 


Institu- 
tion 


10 

4 

15 

19 

15 

11 

38 

38 

17 

25 

5 

3 

100 


Value  of  stock  in  lending  agency 


Under  $150 

$150  to  $249.  .  . 

$250  to  $349 

$350  to  $599 

$600  to  $899 

to  $1,499.  . 
1 ,500  and  over . 

Total 


27 
9 
8 
7 
17 
18 
9 
5 


100 


1 

9 

9 

13 

26 

22 

10 

9 


99 


P2,  except  that  nonreporting  of  demand  deposits  was  even  lower  in  Study 
P3,  for  which  this  asset  served  as  a  primary  validation  source. 

With  regard  to  response  error,  the  significance  of  the  results  is  more 
evident.  On  all  three  types  of  assets,  large  individual  errors  are  frequent. 
Yet  in  the  aggregate  a  clear  tendency  is  apparent  for  these  errors  to 
produce  overstatements  of  the  face  value  of  insurance  and  understate- 
ments of  demand  deposit  balances,  and  to  more  or  less  cancel  out  in 
evaluating  farm  assets.  In  the  case  of  life  insurance,  these  overstatements 
are  beneficial,  serving  to  offset  the  effect  of  nonreporting  so  that  the 
sample-based  aggregates  virtually  coincide  with  the  true  aggregates. 
However,  the  understatements  of  reported  demand  deposit  balances  rein- 
force the  effect  of  nonreporting  in  underestimation  of  the  aggregate. 

Here  again  some  tendency  appears  for  small  amounts  to  be  overstated 
and  for  large  amounts  to  be  understated.  In  terms  of  size  distributions, 
however,  this  tendency  —  as  well  as  one  toward  nonreporting  —  is  of 
little  consequence,  the  distributions  compiled  from  the  interview  data 
corresponding  closely  with  the  true  distributions.  The  principal  effect 
of  these  errors  shows  up  in  the  consistent  underestimates  of  the  measures 
of  sampling  variations,  these  measures  and  the  associated  confidence 
intervals  being  understated  as  much  as  40  percent. 


158     The  Reliability  of  Consumer  Reports  of  Financial  Assets  and  Debts 


Table  47.  Reliability  of  Estimates  of  Sampling  Variability 

in  Estimates  of  the  Means  of  Holdings  of  Different 

Farm  Assets^  Study  P3 


Asset 


(1) 

Acreage  (owners) 

Acreage  (renters) 

Automobiles  and  trucks. 

Other  machinery 

Hogs 

Cattle 

Other  livestock 

Crops 

Stock  in  lending  agency 


Standard  error  of 

the  mean 

■ 

k  =  e/Oy 

r 

Ap- 

Error- 

parent 

free 

(2) 

(3) 

(4) 

(5) 

(6) 

-.09 

-.22 

26 

26 

25 

-.49 

-.64 

18 

20 

22 

1.32 

-.29 

182 

301 

137 

.38 

-.23 

741 

793 

637 

-1.84 

-.79 

246 

516 

401 

-.10 

.01 

939 

944 

758 

-.39 

-.71 

53 

57 

71 

-.99 

-.55 

367 

517 

369 

-1.97 

-.42 

93 

206 

89 

True 
probability 

of  .95 

confidence 

interval 

(7) 


.95 
.92 
.74 
.93 
.55 
.95 
.93 
.83 
.49 


Investigation  of  determinants  of  response  error,  feasible  only  for  life  in- 
surance, suggests  that  —  as  with  other  holdings  ■ — •  high  incomes  were  not 
associated  with  nonreporting  or  with  high  response  error.  Indeed,  in  the 
latter  case,  high  income  was  more  consistently  associated  with  low  response 
errors  than  with  large  response  errors.  These  results  also  support  earlier 
findings  that  nonreporting  rises  with  age;  that  accuracy  of  reporting 
increases  with  use  of  records,  with  level  of  education,  and  with  conduct  of 
the  interview  at  one's  office  instead  of  in  the  home;  and  that  the  use  of  a 
change  form  produces  less  reliable  data  than  the  use  of  a  holdings  form 
(the  latter  based  on  the  analysis  of  errors  in  demand  deposit  balances). 

All  things  considered,  the  accuracy  of  the  reporting  of  the  three  types 
of  assets  studied  in  this  chapter  seems  to  be  considerably  superior  to  that 
of  the  reporting  of  time  deposits  and,  in  many  respects,  to  that  of  debt  as 
well.  Indeed,,  for  various  farm  assets  the  interview  data  would  seem  to  be 
more  reliable  than  the  institutional  data.  Particularly  noteworthy  is  the 
fact  that  reports  of  demand  deposit  balances  are  substantially  more  ac- 
curate than  those  of  time  deposit  balances,  although  most  of  the  dis- 
crepancies between  reported  and  actual  demand  deposit  balances  appear 
to  be  the  result  of  response  errors  rather  than  of  check-float. 


VI.  PANEL  EFFECTS 


Three  major  potential  sources  of  bias  are  recognized  in  the  literature 
on  consumer  panels:  changes  in  population  composition,  panel  mortality, 
and  panel  conditioning  effects.  It  is  because  of  changes  in  population 
composition  that  panel  data  may  become  atypical  of  the  population  even 
though  the  original  sample  may  have  been  perfectly  representative  at  the 
time  it  was  selected.  Most  panels  are  set  up  entirely  at  a  particular 
moment  in  time;  as  a  result,  changes  in  population  composition  are  re- 
flected only  to  a  limited  extent  in  changes  in  the  characteristics  of  the 
panel  (ignoring,  for  the  time  being,  the  effect  of  panel  mortality) . 

Theoretically,  such  factors  could  be  taken  into  account  in  a  continuing 
panel  operation  by  instituting  a  system  of  panel-member  rotation,  and 
such  adjustments  are  included  in  the  large  national  panel  operations  of 
the  United  States  Bureau  of  the  Census  and  the  Market  Research  Corpo- 
ration of  America. 

A  second  source  of  error  is  panel  mortality.  Because  of  the  continuous 
nature  of  a  panel  operation,  nonresponse  tends  to  be  much  higher  than 
on  the  corresponding  single-interview  survey.  Furthermore,  this  non- 
response  is  cumulative,  tending  to  increase  consistently,  although  at  a 
decreasing  rate,  as  time  goes  on.110 

Actually,  studies  made  so  far  indicate  that  bias  due  to  panel  mortality 
may  be  of  much  lesser  consequence  than  has  been  feared.  Thus,  appre- 
ciable mortality  was  obtained  in  a  financial  panel  operated  by  the  Survey 
Research  Center,  in  which  panel  members  were  interviewed  five  times  at 
six-month  intervals.  Yet  little  demographic  bias  resulted  from  panel 
dropouts,  even  though  disinterested  people,  renters,  and  low-income  people 
were  more  likely  to  drop  out.111    In  an  earlier  purchase  survey  in  which 

110  For  example,  see  H.  E.  Allison,  C.  J.  Zwick,  and  A.  Brinser,  "Recruiting 
and  Maintaining  a  Consumer  Panel,"  Journal  of  Marketing,  Vol.  22,  No.  4  (April, 
1958),  pp.  377-90;  Robert  Ferber,  "Observations  on  a  Consumer  Panel  Opera- 
tion," Journal  of  Marketing,  Vol.  17,  No.  3  (January,  1953),  pp.  246-59;  M.  G. 
Sobol,  "Panel  Mortality  and  Panel  Bias,"  Journal  of  the  American  Statistical  Asso- 
ciation, Vol.  54,  No.  285  (March,  1959),  pp.  52-68;  Industrial  Surveys  Company, 
Problems  of  Establishing  a  Consumer  Panel  in  the  New  York  Metropolitan  Area 
(Washington:   United  States  Government  Printing  Office,  1952). 

111  M.  G.  Sobol,  op.  cit. 
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panel  members  were  interviewed  every  month  for  one  year,  no  bias  at  all 
in  demographic  characteristics  was  observed.112  Similarly,  in  a  pilot  oper- 
ation preceding  the  Consumer  Savings  Project,  no  evidence  of  bias  due  to 
panel  mortality  was  observed,  either  in  the  usual  socio-economic  charac- 
teristics of  the  refusals  or  in  their  asset  holdings.113 

To  be  sure,  in  none  of  these  instances  were  validation  data  available. 
Therefore,  the  analysis  of  the  effects  of  panel  mortality  could  be  carried 
out  only  on  such  indirect,  observed  variables  as  respondent  characteristics, 
and  not  on  the  principal  variables  under  investigation. 

The  third  major  source  of  panel  bias,  panel  conditioning  effects,  is 
probably  the  most  pervasive  of  all.  Its  source  derives  from  the  effects  on 
the  panel  members  of  growing  awareness  over  time  of  membership  in  the 
panel.114  Thus,  one  result  of  this  growing  self-consciousness  could  be  a 
tendency  by  the  panel  member  to  modify  his  replies  on  attitudinal  ques- 
tions to  bring  them  more  into  line  with  the  majority  view.  Another  result 
could  be  modification  of  replies  on  later  interviews  to  rationalize  answers 
given  on  earlier  interviews.  Still  another  result  could  emanate  from  loss 
of  interest  in  the  study,  leading  to  the  reporting  of  less  accurate  or  less 
comprehensive  data. 

At  the  same  time,  panel  conditioning  effects  may  have  favorable 
results  as  well.  Thus,  frequent  interviews  have  led  to  improvement  in 
memory,  thereby  producing  more  complete  data  on  past  behavior,  such 
as  on  consumer  purchases.115  In  addition,  the  continuing  contact  between 
the  interviewer  and  the  panel  member  may  reassure  the  latter  of  the 
authenticity  and  reliability  of  the  study,  or  may  prod  his  memory  and 
lead  him  to  provide  more  complete  and/or  more  accurate  information. 

To  date,  evidence  on  panel  conditioning  bias  is  mixed.  Little  condi- 
tioning bias  has  been  observed  on  attitudinal  questions  relating  to  business 
or  economics.116  On  the  other  hand,  a  fair  amount  of  conditioning  bias 
was  observed  in  a  food  purchase  panel,  this  bias  serving  to  increase  pur- 
chases with  time.117  Panel  conditioning  effects  were  also  responsible  for 
a  marked  improvement  in  the  accuracy  in  reporting  durable  goods  pur- 

112  Ferber,  "Observations  on  a  Consumer  Panel  Operation,"  op.  cit. 

113  Robert  Ferber,  Collecting  Financial  Data  by  Consumer  Panel  Techniques, 
Studies  in  Consumer  Savings,  No.  1  (Urbana,  Illinois:  University  of  Illinois 
Bureau  of  Economic  and  Business  Research,  1959),  Chapter  III. 

114  A  summary  of  early  studies  on  the  effect  of  repeated  interviews  on  sample 
results  can  be  found  in  F.  L.  Ruch,  "Effects  of  Repeated  Interviewing  on  the 
Respondent's  Answers,"  Journal  of  Consulting  Psychology,  Vol.  5,  No.  4  (July- 
August,  1941),  pp.  179-82. 

115  Ferber,  "Observations  on  a  Consumer  Panel  Operation,"   op.  cit. 

116  C.  H.  Sandage,  "Do  Research  Panels  Wear  Out?"  Journal  of  Marketing, 
Vol.  20,  No.  4  (April,  1956),  pp.  397-401 ;  M.  G.  Sobol,  ibid. 

117  G.  G.  Quackenbush  and  J.  D.  Shaffer,  Collecting  Food  Purchase  Data  by 
Consumer  Panel   (East  Lansing,  Michigan:    Michigan  State  University,   1960). 
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chases  in  a  monthly  panel  study.118  In  addition,  the  pilot  panel  study  for 
the  Consumer  Savings  Project  produced  considerable  evidence  of  condi- 
tioning effects,  due  primarily  to  consistent  improvements  in  rapport 
through  most  of  the  panel  operation.119  This  improvement  in  rapport 
was  reflected  in  a  sharp  decline  in  refusals  of  dollar  amounts,  in  an 
increase  in  the  number  of  panel  members  supplying  financial  data  from 
records,  and  in  pickup  of  a  fair  amount  of  financial  assets  not  reported 
on  earlier  interviews  although  owned  at  that  time.  Some  panel  members 
did  become  less  cooperative  over  time,  but  this  number  was  small  relative 
to  those  whose  cooperativeness  increased. 

From  the  foregoing,  it  is  clear  that  panel  conditioning  effects  may 
affect  sample  data  for  better  or  for  worse  and  may  do  so  in  numerous 
ways.  In  the  present  state  of  knowledge,  it  is  difficult  to  predict  the  result 
of  such  effects  in  a  particular  study. 

In  what  follows,  our  primary  concern  will  be  with  panel  mortality 
and  with  panel  conditioning  effects.  Because  of  the  absence  of  exact  data 
and  the  short-run  nature  of  these  panel  studies,  the  investigation  of  bias 
due  to  changes  in  population  composition  is  hardly  feasible.  Moreover, 
the  latter  source  of  bias  is  likely  to  be  of  little  consequence  compared  with 
the  two  other  sources. 

The  general  approach  is  to  examine  first  some  overall  effects  of  the 
panel  aspects  on  sample  composition  and  on  respondent  cooperativeness 
in  each  of  the  panel  studies  of  the  Consumer  Savings  Project.  This 
includes  not  only  the  panel  studies,  PI  to  P3,  but  also  the  reinterview 
study,  S3.120  Because  of  the  reinterview  aspect,  the  latter  study  may  be 
considered  as  roughly  equivalent  to  a  two-wave  panel  study,  with  inter- 
views nine  months  apart.  In  each  of  these  studies,  the  data  permit  us  to 
focus  on  the  influence  of  three  aspects  of  the  panel  operation:  changes 
in  the  composition  of  the  panel,  variations  in  the  apparent  cooperativeness 
of  the  panel  members,  and  the  extent  of  pickup  of  panel  member  holdings 
not  reported  previously. 

Second,  we  may  use  the  validation  data  to  investigate  the  nature  of 
response  and  nonresponse  error  in  each  of  these  studies,  as  will  be  done  in 
the  final  part  of  this  chapter.  Such  an  analysis  can  be  carried  out  only 
for  those  holdings  for  which  primary  validation  data  are  available,  which 

118  Ferber,  "Observations  on  a  Consumer  Panel  Operation,"  op.  cit. 

119  Ferber,  Collecting  Financial  Data  by  Consumer  Panel  Techniques,  op.  cit. 

120  One  other  panel  operation  of  the  project,  that  conducted  in  Chicago  in 
1956-57,  is  not  included  here  because  the  circumstances  under  which  it  was  con- 
ducted do  not  make  the  data  comparable  with  those  of  the  other  operations.  Mem- 
bers of  that  sample  were  interviewed  initially  as  part  of  the  larger  one-interview 
study,  and  only  later  were  they  approached  again  to  participate  in  panel  operations. 
The  full  study  is  summarized  in  Ferber,  ibid. 
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is  usually  not  more  than  one  asset  or  debt  in  each  case.  Nevertheless, 
these  data  throw  considerable  light  on  the  dynamic  aspects  of  such  errors, 
especially  since  this  is  the  first  time  such  an  analysis  is  feasible. 

Changes  in  Panel  Composition 

The  response  experience  on  all  four  panel  operations  is  summarized 
in  Table  48.  In  all  instances  appreciable  mortality  is  evident,  as  shown 
by  the  last  column  of  figures  in  this  table.  Roughly  half  of  the  panel 
members  were  lost  by  the  end  of  the  first  three  operations.  Study  P3 
was  much  more  successful,  with  nearly  three-fourths  of  the  panel  members 
still  participating  at  the  end  of  the  fifth  wave.  As  has  been  noted  previ- 
ously, in  terms  of  response  this  operation  was  highly  successful  —  much 
more  so  than  any  of  the  others. 

The  data  in  Table  48  indicate  that  in  each  of  these  four  studies 
refusal,  rather  than  failure  to  contact  sample  members,  was  the  primary 
reason  for  sample  mortality.  This  was  even  true  of  Study  S3,  although 
the  attempt  to  avoid  use  of  names  led  to  a  much  higher  rate  of  noncon- 
tacts  than  that  found  in  the  other  studies.  For  these  four  studies  non- 
contracts  and  dropouts  because  of  ineligibility  were  virtually  negligible. 

The  large  majority  of  refusals  occurred  on  the  first  three  waves  of 
interviews.  By  that  time,  70  percent  to  80  percent  of  the  refusals  en- 
countered over  all  five  waves  had  already  been  obtained.  As  is  evident 
from  the  following  tabulation  of  refusal  rates  overtime  for  each  of  the 
longer  panel  operations,  nearly  half  of  the  refusals  were  obtained  in  the 
first  wave  and  another  fairly  large  proportion  in  the  next  two  waves,  with 
the  refusal  rate  declining  sharply  thereafter. 

Cumulative  refusal  rate 
Wave  Study  PI       Study  P2       Study  P3 

1 20%  20%  10% 

2 27  31  12 

3...' 42  36  16 

4 47  47  18 

5 52  49  20 

The  refusal  rates  in  the  first  three  waves  of  Study  P2  would  have  been 
much  higher  had  it  not  been  for  retention  until  the  fourth  wave  of  sample 
members  who  were  not  giving  financial  information. 

To  what  extent,  if  any,  were  those  who  dropped  out  of  the  panel 
operations  different  from  those  who  remained?  In  view  of  the  varying 
quantities  of  information  available,  it  seems  desirable  to  attempt  to  answer 
this  question  by  comparing  first  those  who  dropped  out  in  the  first  wave 
with  those  who  remained  in  the  first  wave,  and  second,  those  who  dropped 
out  of  later  waves  with  those  who  remained  in  the  panel  to  the  end. 
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Table  48.  Response  and  Nonresponse  Data  on  Four  Panel  Studies 


Characteristic 

Wave 

All 

1 

2 

3 

4 

5 

waves 

Study  S3 

Initial  sample  size 

Interviewed 

Refused 

Noncontact,  still  eligible 

Noncontact,  no  longer  eligible. 

Contact  rate"  (percent) 

Response  rateb  (percent) 

Mortality  rate0  (percent) 


168 

109 

109 

80 

34 

15 

25 

4 

85 

73 

76 

84 

35 

17 

49 
29 

83 
62 
46 


Study  PI 


Initial  sample  size 

Interviewed 

Refused 

Noncontact,  still  eligible 

Noncontact,  no  longer  eligible. 

Contact  rate"  (percent) 

Response  rateb  (percent) 

Mortality  rate0  (percent) 


296 

251 

244 

178 

155 

210 

187 

178 

154 

131 

60 

20 

44 

16 

13 

25 

44 

8 

6 

11 

1 

14 

2 

92 

94 

91 

96 

93 

77 

79 

75 

91 

91 

22 

9 

21 

13 

15 

296 
131 
137 
11 
17 
91 
49 
56 


Study  P2 


Initial  sample  size 

Interviewed 

Refused 

Noncontact,  still  eligible 

Noncontact,  no  longer  eligible. 

Contact  ratea  (percent) 

Response  rateb  (percent) 

Mortality  rate0  (percent) 


411 

304 

263 

266d 

219 

316 

255 

233 

210 

205 

82 

45 

20 

47* 

7 

13 

3 
1 

10 

9 

3 
4 

97 

97 

96 

97 

97 

79 

85 

92 

82 

97 

23 

16 

11 

21 

7 

411 

205 

198 

3 

5 

98 

51 

50 


Study  P3 


Initial  sample  size 

Interviewed 

Refused 

Noncontact,  still  eligible 

Noncontact,  no  longer  eligible. 

Contact  ratea  (percent) 

Response  rateb  (percent) 

Mortality  rate0  (percent) 


417 

180f 

350 

160d-f 

314 

365 

169 

321 

153 

301 

41 

8 

19 

7 

5 

8 

1 

2 

5 

3 

2 

8 

3 

98 

99 

99 

100 

98 

90 

96 

94 

96 

98 

12 

5 

6 

4 

4 

417 
301 
95 
5 
16 
95 
76 
28 


Contacts 


Initial  sample  size 
b  Interviewed 

Interviewed  plus  refused 
c  Noncontacts  plus  refused 
Initial  sample  size 

d  Includes  25  in  Study  P2  and  5  in  Study  P3  dropped  from  panel  because  of  continued 
refusal  to  provide  financial  data. 

e  Only  half  of  panel  interviewed  on  this  wave. 

'  Roughly  half  of  panel  interviewed  by  mail,  with  personal  interview  follow-up  of  nonre- 
spondents. 
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Comparisons  of  the  first  type  suggest  that  high  response  rates  on  the 
initial  interview  were  more  likely  to  be  obtained  from  young  people, 
from  people  of  American  nationality,  and  from  farmers  having  large 
farms.  On  the  other  hand,  refusals  on  the  initial  interview  were  some- 
what more  frequent  among  renters  than  among  owners,  among  older 
people  than  among  younger  people,  among  small-sized  families,  and 
among  those  of  other  than  American  nationality.  Such  inferences  have  to 
be  treated  with  considerable  caution,  however,  because  relatively  little 
classifying  information  could  be  obtained  on  many  of  the  refusals,  par- 
ticularly on  such  characteristics  as  age  of  the  main  wage  earner  and  farm 
tenure. 

Much  the  same  characteristics  serve  to  differentiate  those  who  are 
likely  to  remain  in  the  panel  from  those  who  are  not.  In  particular,  those 
remaining  in  the  panel  throughout  the  operation  were  more  likely  to  be 
owners  than  renters,  have  homes  valued  between  $10,000  and  $25,000,  be 
in  the  lower  age  groups,  have  medium-sized  families,  be  of  American 
nationality,  and  have  somewhat  smaller  balances  in  their  savings  accounts. 
In  most  instances  the  differences  were  not  large,  but  the  fact  that  they 
occurred  in  all  three  panel  operations  would  seem  to  impart  greater  sig- 
nificance to  them  than  to  other  differences. 

Examination  of  the  relative  frequencies  with  which  people  of  different 
characteristics  remain  in  the  panel,  starting  with  those  interviewed  on  the 
first  wave  (and  for  whom  much  additional  data  are  available),  serves  to 
reinforce  the  earlier  findings.  Thus,  a  higher  proportion  of  younger  people 
than  of  older  people  remained  in  the  panel  to  the  end.  In  addition,  those 
with  more  education  were  more  likely  to  remain  in  the  panel  operation 
(although  this  was  not  true  in  Study  PI) ;  the  retired  and  self-employed 
were  more  likely  to  drop  out  than  other  occupational  groups;  and  there 
was  very  little  tendency  for  the  dropout  rate  to  vary  with  income  levels, 
although  the  dropout  rate  did  tend  to  fall  as  the  value  of  the  financial 
assets  of  the  savings  unit  rose.  To  a  large  extent,  these  latter  findings  may 
represent  interaction  with  age,  since  older  people  tend  to  possess  more 
financial  assets  and  are,  of  course,  more  likely  to  be  retired. 

Thus,  these  data  suggest  that  refusals  and  dropouts  were  not  uniform 
by  socio-economic  characteristics,  but  were  somewhat  higher  among  older 
age  groups,  among  those  with  less  education,  among  those  with  greater 
value  of  assets,  and  among  the  self-employed.  These  results  are  consistent 
with  the  earlier  panel  study  (mentioned  in  footnote  113,  p.  160)  which 
also  showed  that  refusals  and  dropouts  were  more  frequent  among  those 
with  little  education,  among  the  self-employed,  and  among  people  who 
were  older,  although  not  in  the  oldest  age  category. 
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Cooperativeness 


Cooperativeness  has  many  different  facets.  In  this  analysis,  the  coop- 
erativeness of  the  panel  members  over  time  is  studied  in  three  respects: 
the  pattern  of  cooperativeness  over  time  and  the  nature  of  the  changes 
that  occur,  the  extent  to  which  the  degree  of  cooperativeness  varies  by 
different  population  groups,  and  the  reasons  for  changes  in  cooperativeness 
over  time.  These  questions  are  studied  with  the  aid  of  two  different  sets 
of  data  - —  subjective  information  obtained  about  the  interview  situation 
and  objective  data  on  the  same  subject.  The  subjective  information  comes 
from  interviewer  reports  on  each  interview  recorded  on  Interviewer 
Report  Forms  (some  of  which  are  shown  in  Appendix  A),  primarily  in 
the  form  of  rating  of  respondent  attitudes,  such  as  overall  cooperativeness, 
accuracy  of  data  supplied,  and  completeness  of  data  supplied,  as  well 
as  changes  in  these  attitudes  from  one  wave  to  another.  The  objective 
information  relates  also  to  respondent  attitudes  but  is  more  factual  in 
nature,  such  as  reports  on  respondent  use  of  records  and  refusal  to  answer 
questions. 

To  a  large  extent,  these  two  sets  of  data  supplement  each  other, 
throwing  light  on  different  aspects  of  respondent  cooperativeness.  This 
is  especially  so  in  view  of  the  experimental  nature  of  these  operations,  one 
result  of  which  was  asking  different  questions  on  respondent  cooperative- 
ness at  different  stages  of  the  panel  operation. 

Patterns  Over  Time 

Interviewer  ratings  of  various  aspects  of  respondent  cooperation  in 
Study  PI  are  presented  in  Table  49.  Four  of  these  ratings  are  subjective 
and  one  —  use  of  records  —  is  objective.  All  five  of  these  ratings  suggest 
that  cooperation  increased  over  time,  although  the  suggestion  is  stronger 
for  some  characteristics  than  for  others.  On  cooperativeness  and  on  com- 
pleteness, a  clear  tendency  is  present  for  the  percentage  rated  "excellent" 
to  rise  over  time  and  for  the  proportion  rated  "poor"  or  "very  poor"  to 
decline.  On  accuracy,  the  proportion  rated  "excellent"  rises,  but  so  does 
the  proportion  rated  "poor"  or  "very  poor."  Much  the  same  is  true  on 
evidence  of  withholding  data:  relatively  fewer  sample  members  are 
labelled  as  sure  withholders  over  time,  but  relatively  fewer  are  also  rated 
as  clearly  not  withholding  anything.  The  objective  measure  of  use  of 
records  indicates  that  a  much  lower  proportion  of  the  sample  members 
did  not  use  any  records  as  time  went  on,  although  the  proportion  always 
using  records  did  not  increase. 

To  measure  the  relative  sensitivity  of  these  measures  to  changes  in 
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Table  49.  Measures  of  Respondent  Cooperation 
Over  Time,  Study  PI 


Wave 

Base 

Percentage  of  total  respondents 
rated 

Characteristic 

Ex- 
cellent 

Good 

Fair 

Poor  or 
very 
poor 

Degree  of  cooperativeness 

1 
2 

199 
182 

52 
62 

29 
26 

13 
10 

7 
3 

3 

171 

64 

24 

8 

3 

4 

71 

63 

23 

6 

9 

5 

137 

68 

20 

9 

4 

Degree  of  accuracy 

1 
2 

195 
181 

36 
37 

28 
25 

18 
20 

7 
6 

3 

170 

40 

29 

14 

7 

4 

71 

38 

28 

13 

6 

5 

137 

43 

31 

13 

12 

Degree  of  completeness 

1 
2 

194 
181 

37 
45 

32 
23 

16 
18 

14 
15 

3 

170 

45 

30 

12 

12 

4 

71 

44 

27 

8 

21 

5 

1 

2 

137 

101 
181 

50 

28 

15 

7 

Always 

Occasionally 

Never 

Use  of  records 

29 
29 

27 

27 

45 

44 

3 

173 

20 

36 

43 

4 

71 

11 

51 

38 

5 

1 
2 

137 

201 
183 

28 

41 

31 

None 

Some 

Clear 
evidence 

Evidence  of  withholding  data. . . . 

70 
69 

9 
9 

21 
22 

3 

172 

61 

17 

22 

4 

71 

55 

25 

20 

5 

137 

64 

22 

14 

respondent  cooperation  over  time,  the  following  so-called  Index  of  Sen- 
sitivity (S)  will  be  used: 

2>« 


Pu- 


200 


2^1  \P%t  —  Pi,t-1 

20(3 


where 


Pa  is  the  proportion  of  sample  members  at  time  t  rated  in  class  i, 
Pi,t-i  is  the  proportion  of  sample  members  at  time  t-\  rated  in  class  z,  and 
n  is  the  number  of  ratings,  i.e.,  i  =  1, .  .  . ,  n. 


Panel  Effects 


167 


Table  50.  Ratings  of  Change  in  Respondent  Cooperation 
Since  Last  Interview,  Study  PI 


Characteristic 


Waves 


Base 


Percentage  rated 


Better 


Same        Worse 


Degree  of  cooperativeness . 


Degree  of  accuracy. 


Degree  of  completeness . 


2  vs.  1 

3  vs.  2 

4  vs.  3 

5  vs.  4 

2  vs.  1 

3  vs.  2 

4  vs.  3 

5  vs.  4 

2  vs.  1 

3  vs.  2 

4  vs.  3 

5  vs.  4 


150 

64 

100 

89 
150 

63 
100 

89 
150 

63 
100 


30 

19 

9 

12 

30 
20 
13 
12 

27 
22 
11 
15 


68 
73 
83 
80 

66 

69 

71 

77 

71 
70 
75 
74 


3 

9 

16 

11 


14 
11 


The  value  of  the  first  term  on  the  left  can  vary  between  200/n  and 
zero.  Hence,  dividing  by  200/n  yields  a  standardized  ratio  that  varies 
between  zero  and  one.  The  higher  is  the  value  of  S,  the  greater  is  the 
overall  change  in  cooperation  by  that  characteristic.  However,  the  direc- 
tion of  the  change  has  to  be  inferred  from  the  data. 

Applying  this  measure  to  the  data  in  Table  49  for  the  first  and  last 
waves  yields  the  following  values  of  S : 


Degree  of  cooperativeness .  .  .  . 

Degree  of  accuracy 

Degree  of  completeness 

Use  of  records 

Evidence  of  withholding  data . 


.15 

.10 

.125 

.145 

.13 


These  results  suggest  that  ratings  of  cooperativeness  and  use  of  records 
have  noticeably  more  of  what  might  be  termed  discriminatory  power  than 
the  other  ratings.  Moreover,  both  of  these  ratings  indicate  that  the  co- 
operativeness of  sample  members  increased  over  time.  This  inference 
receives  further  support  from  the  data  in  Table  50,  based  on  interviewer 
ratings  of  change  in  respondent  attitudes  from  one  wave  to  another  for 
three  of  the  same  characteristics  covered  in  Table  49.  Particularly  on  the 
earlier  waves,  the  proportion  of  sample  members  rated  as  improving  is 
seen  to  be  substantially  higher  than  the  proportion  becoming  worse,  for 
each  of  these  three  characteristics.  Over  time  the  proportion  of  the  sample 
improving  by  any  of  these  three  ratings  declines  while  the  proportion 
becoming  worse  rises.  In  part  this  trend  is  a  statistical  phenomenon, 
for  with  increasing  proportions  of  sample  members  being  rated  "excellent" 
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Table  51.  Dropout  Rates  by  Measures  of  Respondent 
Cooperation ,  Study  PI 


Characteristic 

Wave 

Percentage  with  given  rating 

on  previous  wave  dropping  out  on 

current  wave 

Ex- 
cellent 

Good 

Fair 

Poor 

Very 
poor 

Degree  of  cooperativeness 

2 
3 
4 
5 

2 
3 
4 
5 

2 
3 
4 
5 

2 
3 
4 
5 

2 
3 
4 
5 

10 
5 
5 

16 

11 
3 
3 
9 

8 

2 

4 

13 

21 

15 

20 

6 

7 

4 

12 

20 

13 

7- 

16 

11 

24 
39 
36 
0b 

23 

24 

5 

11 

25 
34 
10 
14 

33* 

67b 

0b 

0b 

37 
20 
28 
45 

33 
24 
17 
54 

50a 

Degree  of  accuracy 

0b 
75b 
75b 

67a 

Degree  of  completeness 

33a 

67 
100b 

44 

33b 

67a 

100b 

Always 

Occasionally 

Never 

Use  of  records 

14 
2 
3 

12 

7 

8 

5 

11 

29 

21 
25 
44 

None 

Some 

Clear 
evidence 

Evidence  of  withholding  data 

10 
7 
5 

13 

21 
19 
10 
22 

38 
25 
26 
57 

a  Based  on  5  to  9  observations. 

b  Based  on  fewer  than  5  observations. 


on  an  absolute  basis  (Table  49),  the  proportion  that  could  be  rated  better 
in  a  temporal  sense  is  bounded  more  at  the  upper  limit  than  at  the  lower 
limit.  In  part,  too,  the  increasing  proportion  rated  "worse"  does  suggest 
less  cooperation  on  the  part  of  some  of  the  sample  members. 

That  these  ratings  do  reflect  the  cooperation  tendencies  of  the  panel 
members  is  brought  out  by  the  data  in  Table  51,  showing  what  proportion 
of  those  given  a  particular  rating  on  one  wave  dropped  out  on  the  next 
wave.  For  all  five  characteristics  and  for  all  waves  a  strong  tendency  is 
apparent  for  the  dropout  rate  to  rise  as  the  rating  becomes  worse.  If 
anything,  this  tendency  seems  to  become  more  pronounced  over  time,  with 
the  spread  between  dropout  rates  at  the  extremes  rising  from  Wave  2  to 
Wave  5  for  each  of  the  five  characteristics  shown,  although  the  sample 
sizes  for  many  of  the  poorer  ratings  are  very  small. 


Panel  Effects  169 

Much  the  same  results  were  obtained  with  Study  P2  as  with  Study  PI. 
Cooperation  improved  generally  over  time  for  all  the  characteristics  on 
which  ratings  were  obtained.  Since  ratings  were  not  obtained  in  most 
instances  on  so  continuous  a  basis  as  on  Study  PI,  it  is  difficult  to  compare 
the  sensitivity  of  these  various  measures.  However,  some  comparisons  are 
possible,  and  in  those  cases  the  following  values  of  S  were  obtained  for 
Study  P2 : 

Waves                                              Characteristic  S 

2  to  5 Interest  in  study  .  1 1 

Attitude  toward  giving  information  .04 

Use  of  records  .  1 5 

1  to  4 Accuracy  on  checking  accounts  .  08 

Accuracy  on  savings  accounts  .08 

Accuracy  on  life  insurance  .  24 

Accuracy  on  corporate  stock  .  18 

Completeness  on  checking  accounts  .  065 

Completeness  on  savings  accounts  .  12 

Completeness  on  life  insurance  .  12 

Completeness  on  corporate  stock  .  02 

As  before,  use  of  records  turns  out  to  have  relatively  high  sensitivity, 
the  highest  of  the  three  ratings  with  which  it  can  be  compared.  However, 
even  greater  sensitivity  is  obtained  with  the  accuracy  ratings  on  life  in- 
surance and  on  corporate  stock.  In  the  former  case  in  particular,  sub- 
stantial improvement  in  both  accuracy  and  completeness  of  the  data  is 
reported.  Why  this  should  be  more  characteristic  of  life  insurance  than 
of  any  of  the  other  three  assets  is  not  clear.  Possibly  to  the  extent  that 
sample  members  became  more  cooperative,  they  may  have  been  more 
likely  to  report  fully  on  a  relatively  insensitive  asset  such  as  life  insurance 
—  or  it  may  have  seemed  that  way  to  the  interviewer.  If  so,  however,  the 
value  of  S  for  the  checking  account  ratings  should  not  be  much  less  than 
for  those  of  life  insurance,  but  this  is  clearly  not  true. 

The  ratings  of  change  in  cooperation  on  Studies  P2  and  P3  exhibit 
much  the  same  pattern  as  those  on  Study  PI.  The  proportions  improving 
exceeded  the  proportions  becoming  less  cooperative  on  every  wave, 
although  the  difference  between  the  two  sets  of  proportions  declined  over 
time.  At  the  same  time,  the  ratings  became  more  stable  over  time,  much 
more  so  than  in  Study  PI.  By  the  last  wave,  roughly  85  percent  of 
the  sample  members  on  either  rating  in  Study  P2  did  not  shift  in 
cooperativeness. 

The  meaningfulness  of  the  ratings  is  again  supported  by  comparisons 
of  the  proportions  of  sample  members  with  given  ratings  on  one  wave 
dropping  out  on  the  next  wave.  These  comparisons  show  that,  by  any 
characteristic  rated,  a  much  higher  proportion  of  those  rated  low  drop 
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Table  52.   Refusal  Rate  for  Selected  Holdings, 
by  Wave  and  by  Study 

(percentage  of  known  owning  savings  units  refusing  amounts) 


Wave 


Savings 
accounts 


Stocks 
and  non- 
United  States 
government 
bonds 


Life 
insurance 


Ten  assets 
and  debtsa 


Constant 
sample:    stocks 

and  non- 
United  States 
government 
bonds 


Study 

PI 

1 

2 

3 

27.8 

25.4 

20.2 

8.9 

38.2 

32.2 
13.8 

4.1 

20.8 

23.5 

5.6 

3.8 

25.7 
23.5 
13.5 

7.9 

29.4 

30.0 

9  9 

5 

4.5 

Study 

P2 

1 

13.2 

13.3 

15.0 

10.6 

4.5 

2 

17.1 

13.3 

6.5 

10.6 

5.1 

3 

14.3 

14.9 

9.1 

10.1 

b 

5 

6.6 

1.1 

1.6 

2.0 

0 

Study 

P3 

1 

14.1 

14.3 

8.8 

7.0 

2.2 
3.8 
8.2 
5.2 

1.3 

0.0 

3.1 

.7 

4.0 
3.1 
4.8 
2.5 

5  0C 

2 

5  0C 

3 

5 

b 

4.6C 

a  Includes  checking  accounts,  savings  accounts,  government  bonds,  corporate  stock  and  non- 
United  States  government  bonds,  life  insurance,  brokerage  accounts,  annuities,  mortgages,  and  other 
non-installment  debt. 

^Savings  units  are  counted  only  if  dollar  holdings  (not  change)  were  requested.  _  Wave  4 
is  omitted  because  all  savings  units  were  asked  only  for  change  on  that  wave;  the  same  is  true  of 
the  constant  sample  on  Wave  3  of  Studies  P2  and  P3. 

c  Savings  accounts,  rather  than  stocks  and  nongovernment  bonds. 


out  on  the  next  wave  than  of  those  rated  high.  As  before,  the  difference 
between  dropout  rates  at  high  and  low  ratings  tends,  if  anything,  to 
increase  over  time. 

A  final,  and  fairly  objective,  measure  of  the  pattern  of  respondent 
cooperation  over  time  is  the  trend  in  the  refusal  rate  for  selected  holdings. 
This  rate  is  defined  as  the  proportion  of  sample  members  interviewed  on 
a  particular  wave  who  refuse  to  give  dollar  amounts  of  their  holdings 
when  asked.121  It  is  not  an  overall  measure  of  respondent  cooperation  be- 
cause the  base  necessarily  excludes  nonreporters  and  because  respondents 
could  be  apparently  cooperative  but  supply  misleading  information.  It  is 
essentially  a  supplementary  measure  of  cooperativeness. 

These  refusal  rates  are  shown  in  Table  52  for  all  three  studies  for 


121  Except  for  corporate  stock  and  bonds,  in  which  cases  the  refusal  refers  to 
the  name  and  number  of  the  holding. 
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selected  individual  assets  and  for  a  group  of  10  holdings,  as  listed  at 
the  bottom  of  that  table.  According  to  these  data,  refusal  rates  declined 
sharply  in  virtually  all  instances,  for  both  individual  holdings  and  for 
the  group  of  10  holdings.  Moreover,  similar  declines  are  apparent  for 
stocks  and  nongovernment  bonds  in  Studies  PI  and  P2  and  for  savings 
accounts  in  Study  P3,  as  shown  in  the  last  column  of  Table  52.  In  this 
sense,  therefore,  cooperativeness  appears  to  have  risen  sharply  during  all 
three  panel  operations,  and  to  a  large  extent  because  of  improved  rapport 
with  respondents  remaining  in  the  panel. 

Cooperation  by  Respondent  Characteristics 

Data  on  the  extent  to  which  degree  of  cooperation  varied  with  a 
number  of  respondent  characteristics  are  summarized  in  Table  53  for  the 
three  panel  operations.  A  plus  sign  indicates  a  noticeable  positive  relation- 
ship between  higher  values  of  the  respondent  characteristic  and  reliability 
of  the  data  by  the  particular  rating  involved,  a  negative  sign  indicates  a 
reverse  relationship,  and  a  zero  shows  absence  of  any  noticeable  relation- 
ship.122 For  those  respondent  characteristics  that  do  not  follow  any 
numerical  order,  that  category  of  the  characteristic  for  which  the  inter- 
viewer rating  is  highest  is  shown  in  the  tables.  To  facilitate  comparability 
among  ratings,  the  same  directional  system  of  notation  has  been  used  for 
the  ratings  of  uncooperativeness,  such  as  "evidence  of  withholding  of 
data."  In  other  words,  in  those  instances  a  positive  sign  indicates  a  posi- 
tive association  with  what  might  be  termed  "data  reliability,"  meaning  a 
negative  association  with  the  frequency  of  the  particular  rating. 

For  Study  PI,  Table  53  shows  that  data  reliability  tends  to  increase 
with  size  of  the  savings  unit,  with  education,  with  the  amount  in  savings 
accounts,  and  with  the  "blue  collarness"  of  the  occupation.  Reliability 
tends  to  decline  with  age  and  possibly  with  income  and  does  not  appear 
to  be  associated  with  value  of  home  and  with  gross  value  of  (financial) 
assets.  However,  the  data  relating  to  Study  P2  do  not  exhibit  the  same 
patterns.  In  that  study,  reliability  appears  to  be  associated  positively  not 
only  with  education  and  with  blue-collar  occupations,  the  same  as  in  Study 
PI,  but  also  with  American  nationality,  which  was  not  the  case  in  the 
previous  study.  Gross  value  of  assets  exhibits  a  negative  relationship  with 
data  reliability,  but  no  such  relationship  was  shown  in  Study  PI. 

The   data  for  the  farm  panel  present   perhaps   the   least-consistent 

122  A  "noticeable  relationship"  is  admittedly  somewhat  arbitrary.  In  practice, 
the  criteria  for  establishing  such  a  relationship  were  either  a  monotonic  trend  in 
the  proportions  with  extreme  ratings  or  a  sharp  drop  ( 1 0  percentage  points  or 
more)  from  one  end  of  a  respondent  characteristic  to  the  other.  Since  the  basic 
data  are  presented  in  Appendix  B,  the  reader  is  free  to  modify  these  definitions  as 
he  wishes. 
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Table  53.  Relationship  Between  Respondent  Characteristics  and 
Cooperation  Rating,  Studies  PI  to  P3 


Study  PI 


Respondent 
characteristic 


Coopera- 
tiveness 


Accuracy 


Com- 
pleteness 


Use  of 
records 


With- 
holding 
dataa 


Value  of  home 

Age  of  main  wage  earner. . 

Size  of  savings  unit 

Nationality 

Occupation 

Education 

Income 

Gross  value  of  assets 

Balance  in  savings  accounts 


+ 

+ 
Other 
Labor 

+ 
+ 

+ 
+ 


0 

0 
Other 
Labor 

+ 

0 

+ 


+ 

0 

Labor 

0 

+ 
0 

+ 


0 

0 
U.S. 
Labor 

0 

+ 
+ 


0 

+ 

0 

Labor 

+ 


+ 


Study  P2 


Respondent 
characteristic 

Use  of 
records 

Willingness  to  give 
financial  information 

Accuracy  of 
savings  accounts 

0 

+ 

U.S. 

Labor 

+ 

0 

0 

0 
0 
0 
U.S. 
Self-employed 
+ 
+ 

0 

+ 

Age  of  main  wage  earner. . .  . 

Size  of  savings  unit 

Nationality 

Occupation 

Education 

Income 

Gross  value  of  assets 

Balance  in  savings  accounts .  . 

+ 

U.S. 

Labor,  professional 

+ 

0 

Study  P3 


Respondent 

Checking  account 
balances 

Data 

withheldb 

Lack  of 
knowl- 
edge15 

Use  of 

characteristic 

Accuracy 

Com- 
pleteness 

records 

0 

U.S. 
Owner 

0 
0 
0 

0 

+ 

U.S. 
Owner 

0 
0 
0 

0 

+ 

0 
Other 
Tenant 

+ 

+ 

+ 

+ 

0 
Tenant 

+ 

0 

+ 
+ 

+ 

Age  of  farmer 

Size  of  savings  unit 

Nationality 

Tenure 

Education 

Income 

0 

0 

Tenant 

+ 

0 

Size  of  farm 

+ 

Amount  of  farm  debt 

Balance  in  savings  accounts .  . 

0 
0 

a  Relationship  is  with  improved  reliability  of  data.  For  example,  a  negative  sign  for  "age 
of  wage  earner"  means  that  as  age  increased,  withholding  of  data  was  suspected  more  frequently, 
i.e.,  the  data  seemed  less  reliable. 

b  Relationship  is  with  improved  reliability  of  data.  For  example,  a  plus  sign  under  "data 
withheld"  for  "condition  of  home"  means  that  as  the  condition  of  the  home  improved,  instances 
of    data   withheld   became    relatively   less   frequent,    i.e.,    the    data   appeared   to   be    more    reliable. 
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picture  of  all.  In  only  one  instance  are  as  many  as  four  of  the  ratings 
consistent  with  each  other.  Judging  by  the  instances  in  which  three  or 
more  ratings  coincide,  there  is  evidence  of  a  positive  association  between 
data  reliability  and  condition  of  home,  education,  size  of  farm,  and  tenant 
status.  Tendency  to  cooperate  appears  to  decline  with  age  of  the  farmer, 
income,  and  size  of  the  savings  unit;  but  no  relationship  appears  to  exist 
with  nationality,  value  of  assets,  and  amount  in  savings  accounts.  How- 
ever, these  broad  statements  have  to  be  hedged  by  the  fact  that  Table  53 
in  many  ways  exhibits  two  different  sets  of  relationships,  one  set  for  the 
ratings  of  checking  account  balances  and  the  other  set  for  the  other  three 
more  general  ratings.  This  is  brought  out  when  we  compare  the  two  sets 
of  ratings  by  characteristic,  using  plus,  minus,  and  zero  signs  only  when 
all  the  ratings  in  a  set  agree  with  each  other.  The  result  is: 

Checking 
account  ratings        Other  ratings 

Condition  of  home —  + 

Age  of  farmer ....  — 

Size  of  savings  unit —  .... 

Nationality United  States  .... 

Tenure Owner  Tenant 

Education ....  + 

Income 0  — 

Value  of  assets 0  .... 

Size  of  farm ....  + 

Amount  of  farm  debt —  .... 

Balance  in  savings  accounts 0  .... 

On  this  basis,  checking  account  ratings  are  seen  to  be  negatively  corre- 
lated with  condition  of  home,  with  size  of  savings  unit,  and  with  the 
amount  of  farm  debt;  to  be  positively  correlated  with  American  nation- 
ality and  with  ownership  of  the  farm;  and  to  show  no  relationship  to 
income,  value  of  assets,  and  balance  in  savings  accounts.  On  the  other 
hand,  the  more  general  ratings  indicate  positive  correlation  with  condition 
of  home,  with  education,  with  tenant  status,  and  with  size  of  farm;  and 
negative  correlation  with  age  of  farmer  and  with  income.  Clearly,  in 
three  of  these  instances  the  two  sets  of  ratings  conflict  with  each  other  — 
in  condition  of  home,  in  tenure  status,  and  in  income.  In  addition,  Table 
53  suggests  that  three  more  conflicts  may  exist,  namely,  in  size  of  savings 
unit,  in  amount  of  farm  debt,  and  in  balance  in  savings  accounts. 

Altogether,  the  two  sets  of  ratings  appear  to  disagree  much  more  often 
than  they  agree.  However,  they  may  well  be  measuring  different  things, 
the  checking  account  ratings  focusing  on  a  specific  asset  and  the  other 
ratings  providing  a  more  general  indication  of  the  reliability  of  the  data. 
If  so,  this  suggests  that  overall  ratings  may  camouflage  a  heterogeneous 
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pattern  of  data  reliability  in  the  interview  and  that  a  respondent  may 
provide  reliable  data  on  one  set  of  questions  but  not  on  another.  General 
ratings  may  still  be  useful  since,  as  has  been  shown  in  the  previous  section, 
such  ratings  do  serve  to  reflect  respondents'  propensity  to  drop  out  of  a 
panel  operation.  However,  for  analyzing  the  reliability  of  data  on  indi- 
vidual holdings  it  would  seem  desirable  to  secure  ratings  for  each  holding 
separately. 

From  a  more  general  point  of  view,  the  data  in  this  section  suggest 
that  respondent  cooperation  does  tend  to  vary  with  particular  character- 
istics. Thus,  taken  together,  these  studies  show  that  respondent  coopera- 
tion tends  to  increase  with  education  and  with  laboring  occupations  and 
to  decline  with  age  and  possibly  with  income.  Perhaps  equally  significant 
is  the  fact  that  there  is  relatively  little  evidence  to  suggest  that  respondent 
cooperation  is  influenced  by  the  gross  value  of  assets  or  by  the  size  of 
savings  account  balances,  particularly  since  Study  PI  had  basically  a 
higher  income  sample  than  Study  P2. 

Reasons  for  Change  in  Cooperativeness 

Did  the  improvement  in  panel  cooperativeness  over  time  result  pri- 
marily from  a  change  in  the  sentiments  of  continuing  panel  members  or 
was  it  a  manifestation  of  differential  panel  mortality?  Circumstantial 
evidence  from  data  presented  in  preceding  sections  suggests  the  latter 
explanation.  Dropouts  were  more  frequent  among  older  people,  among 
those  with  less  education,  among  the  retired  and  the  self-employed,  and 
among  farm  owners.  At  the  same  time,  Table  53  shows  that,  by  and  large, 
these  were  the  groups  to  receive  poor  cooperativeness  ratings  initially. 
Hence,  if  these  groups  contain  more  dropouts  than  others,  it  seems  plau- 
sible to  infer  that  higher  frequency  of  panel  mortality  among  the  less 
cooperative  is  the  principal  reason  for  the  improvement  in  cooperativeness 
over  time. 

Such  an  inference  cannot  be  conclusive,  however,  since  the  possibility 
remains  that  many  of  the  dropouts,  even  in  these  particular  groups,  were 
not  less  cooperative  than  other  panel  members.  In  other  words,  com- 
paring marginal  distributions  at  different  points  in  time,  as  was  done 
earlier  in  this  chapter,  cannot  provide  a  definitive  answer  to  this  question. 
Such  an  answer  can  only  be  obtained  by  studying  joint  distributions  over 
time.  Distributions  of  this  type  for  a  number  of  cooperativeness  ratings 
have  been  obtained  by  preparing  "tree  diagrams"  in  which  the  coopera- 
tiveness rating  of  each  panel  member  was  traced  from  one  wave  to 
another,  as  shown  in  Figure  2  for  the  first  three  waves  of  Study  PI. 

The  numerals  in  this  diagram  show  how  many  respondents  received 
different  ratings  on  each  wave  and  how  these  ratings  changed  from  one 


Panel  Effects 


175 


Figure  2.   Cooperativeness  :    Waves  1,  2,  and  3,  Study  PI 

WAVE    I  WAVE  2  WAVE   3 


I. 

66 

13 

,5 

1, 

5" 

II. 

2, 

7, 

2, 

0 

1 

III. 

1, 

1, 

2, 

0, 

0 

_  V. 

1, 

1, 

1, 

0, 

1 

I.  5,  6, 

0, 

0,    1  " 

11.  8,  5, 

3, 
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III.  0,  0, 

0, 

0,  1 

IV.  0,  0, 

0, 

l|  0 

Y.   1,  3, 

1, 

0,  0 

0,  0,  0 

2,   I,    I 


Y.  107 


wave  to  another.  For  example,  103  people  were  rated  "excellent"  in  terms 
of  cooperativeness  on  Wave  1.  On  Wave  2,  113  received  the  same  rating, 
this  number  consisting  of  73  rated  "excellent"  on  Wave  1,  24  rated  "good" 
previously,  6  rated  "fair"  previously,  2  rated  "poor"  or  "very  poor,"  and  8 
nonrespondents.  On  Wave  3,  110  people  received  a  rating  of  "excellent," 
the  numerals  on  the  lines  between  Wave  2  and  Wave  3  showing  how  many 
of  these  people  had  the  same  or  different  ratings  on  Wave  2.  The  figures 
in  the  brackets  following  the  Wave  3  figures  show  the  Wave  1  origin  of 
each  group  for  each  rating  on  Wave  3.  For  example,  the  first  row  of  the 
matrix  next  to  1 10  on  Wave  3  indicates  that  of  those  receiving  this  rating 
on  that  wave,  66  were  rated  "excellent"  on  Wave  1,  13  were  rated  "good" 
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on  Wave  1,  5  were  rated  "fair"  on  that  wave,  1  was  rated  "poor"  or  "very 
poor,"  and  5  were  nonrespondents  on  Wave  1.  The  total  is  90,  the  num- 
ber of  the  110  to  receive  a  rating  of  excellent  on  Wave  2.  The  other  rows 
of  this  matrix  and  the  the  other  four  matrixes  are  interpreted  in  a  similar 
manner. 

Additional  waves  are  not  shown  because  of  the  tremendously  compli- 
cated nature  of  these  diagrams.  However,  these  three  waves  alone  are 
sufficient  to  provide  clear  answers  to  the  central  question  of  this  section, 
as  is  evident  from  the  following  inferences  that  may  be  drawn  from  Figure 
2: 

( 1 )  Cooperativeness  among  panel  members  rose  from  Wave  1  to  Wave 
2  and  changed  very  little  on  the  following  wave.  The  relative  marginal 
distributions,  excluding  the  nonrespondents,  are  as  follows: 

Wave  1  Wave  2  Wave  3 

Excellent 52%  62%  64% 

Good 28  26  24 

Fair 13  10  8 

Poor  or  very  poor 7  2  4 

Total 100%  100%  100% 

Base  number 199  182  171 

(2)  Dropouts  on  Wave  2  were  primarily  those  who  were  less  cooper- 
ative on  Wave  1,  and  the  same  phenomenon  carried  over,  to  a  lesser 
extent,  on  Wave  3.  The  following  are  the  percentages  of  sample  members 
given  a  particular  cooperativeness  rating  on  Wave  1  who  dropped  out  on 

Wave  2  or  on  Wave  3 : 

Wave  2         Wave  3 
dropouts         dropouts 

Excellent 13%  5% 

Good 21  11 

Fair 24  16 

Poor  or  very  poor 43  7 

Of  those  with  particular  ratings  on  Wave  2,  the  following  proportions 
dropped  out  on  Wave  3 : 

Excellent 5% 

Good 15 

Fair 26 

Poor  or  very  poor 25 

(3)  Cooperativeness  among  those  interviewed  on  all  three  waves  also 
rose  substantially  from  Wave  1  to  Wave  2  and  retrogressed  slightly  on  the 
next  wave,  as  is  evident  from  the  following  marginal  distributions  for  this 
"constant"  sample: 
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Wave  1  Wave  2  Wave  3 

Excellent 58%  69%  68% 

Good 27  23  21 

Fair 10  6  8 

Poor  or  very  poor 5  1  3 

Total 100%  99%  100% 

Base  number 146  146  146 

(4)  Cooperativeness  among  those  later  dropping  out  was  much  lower 
than  among  panel  members  interviewed  on  all  three  waves,  as  shown  by 
the  distribution  of  these  ratings  on  Wave  1  of  those  dropping  out  on 
Waves  2  or  3 : 

Excellent 34% 

Good 34 

Fair 19 

Poor  or  very  poor 13 

Total 100% 

Base  number 53 

The  conclusion  is  clearly  that  the  increase  in  cooperativeness  over  time 
is  a  product  of  two  factors  —  the  greater  panel  mortality  among  those 
less  cooperative  initially  and  the  increase  in  rapport  among  those  remain- 
ing in  the  panel.  Perhaps  most  surprising  is  the  fact  that  the  increase  in 
cooperativeness  among  those  that  remained  in  the  panel  appears  to  have 
been  at  least  as  large  as  that  brought  about  by  differential  panel  mortality. 

Other  comparisons  based  on  these  three  diagrams  suggest,  however, 
that  in  most  instances  the  differential  panel  mortality  is  primarily  respon- 
sible for  the  increase  in  cooperativeness.  Comparisons  for  terminal  waves 
of  each  panel,  based  on  respondent  distributions  for  both  total  and  constant 
samples,  reveal  that  overall  cooperativeness  improved  in  every  instance. 
In  no  case,  however,  was  the  improvement  in  cooperativeness  of  the  con- 
stant sample  so  pronounced  as  that  of  the  total  sample. 

Supporting  data  on  this  general  point  are  provided  in  Table  54,  which 
shows  the  percentage  of  panel  members  with  particular  ratings  on  Wave 
1  that  had  dropped  out  by  the  terminal  wave.  In  all  instances,  the  drop- 
out rate  correlates  inversely  with  degree  of  cooperativeness,  the  increase 
in  dropout  rates  with  lowered  cooperativeness  being  particularly  substan- 
tial for  Studies  PI  and  P2.  The  principal  cause  of  improved  cooperative- 
ness, therefore,  appears  to  have  been  differential  panel  mortality. 

Pickup 

The  frequency  of  pickup  in  the  three  panel  operations  is  shown  in 
Table  55  for  each  of  10  types  of  holdings.   As  is  evident  from  this  table, 
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Table  54.  Dropouts  as  Proportion  of  Total  Panel  Members, 
by  cooperativeness  ratings  on  earlier  waves 


Characteristic 

Waves 
com- 
pared 

Percentage  rated 

Study 

Ex- 
cellent 

Good 

Fair 

Poor  or 
very 
poor 

PI 

Accuracy  of  report 

Completeness  of  report .  .  . 

Accuracy  of  checking 
accounts 

Accuracy  of  savings 

accounts 

Completeness  of  checking 
accounts 

1  to  5 
1  to  5 

1  to  4 

1  to  4 

1  to  4 

1  to  4 
1  to  4 

1  to  4 

1  to  5 

2  to  5 

30 
25 

27 
34 

46 
41 

59 
64 

Fully 

Fairly 

Not 
much 

Very 
little 

P2 

42 

39 

44 

43 
3 

52 

60 

58 

66 
19 

67 

47 

82 

15 
33 

92 
77 
88 

Completeness  of  savings 

accounts 

Interest  in  study 

Use  of  records 

60 
61 

Always 

Sometimes 

Never 

17 
8 

19 
14 

53 

P3 

Use  of  records 

26 

Willingness  to  give 

financial  information .  .  . 

Ex- 
cellent 

Good 

Fair 

Poor  or 
very 
poor 

4 

8 

22 

25 

pickup  varied  considerably,  not  only  by  holding,  but  also  by  the  particular 
panel  operation.  Thus,  pickup  of  checking  accounts  on  Study  P3  appears 
to  have  been  negligible,  whereas  pickup  of  stocks  and  bonds  on  Study  P2 
amounted  to  as  much  as  36  percent  of  the  total,  in  terms  of  the  number 
of  holdings.  Moreover,  although  pickup  of  the  total  value  of  checking 
accounts  was  very  low  on  Study  P3,  it  had  reached  a  figure  of  almost  18 
percent  of  these  accounts  on  Study  PI. 

Nevertheless,  certain  general  patterns  appear  to  exist  in  these  data. 
Thus,  on  an  overall  basis,  pickup  seems  to  have  been  lower  for  Study  P3 
than  for  either  of  the  other  two  studies.  This  may  have  been  due  to  the 
possibly  higher  interviewer  efficiency  obtained  on  this  last  study  or  to  the 
greater  cooperativeness  of  farmers  as  sample  respondents. 

Also  evident  is  the  fact  that  pickup  was  much  higher  in  value  terms 
than  in  terms  of  the  number  of  holdings  for  Study  PI,  whereas  the  op- 
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Table  55.  Pickup  of  Financial  Holdings  on  Later  Waves,  by  Study 
(percentage  of  total  holdings  picked  up  after  wave  1) 


Number  of  holdings 

Value  of  holdings 

Holding 

Study 
PI 

Study 
P2 

Study 
P3 

Study 
PI 

Study 
P2 

Study 
P3 

Checking  accounts 

7.5 

8.6 

1.7 

18.0 

7.1 

.3 

Savings  accounts 

7.0 

11.2 

7.0 

35.1 

6.7 

10.7 

United  States  government 

savings  bonds 

19.3 

23.1 

5.8a 

5.8 

17.6 

14. la 

Stocks  and  other  bonds 

8.7 

36.2 

11.7 

23.8 

17.3 

4.5 

Life  insurance 

3.8 

16.7 

6.8 

7.4 

15.1 

5.7 

Pension  plan  equity 

Loans  to  others 

0 
14.3 

23.1 
29.4 

b 

5.0 

5.8 
46.3 

.7 
16.9 

b 

7^4 

Real  estate 

2.6 

11.3 

10. 6C 

6.7 

11.4 

.5° 

Mortgages 

Personal  loans 

4.9 
30.6 

12.7 
22.2 

7.0d 
5.3e 

4.1 
43.1 

14.3 
8.8 

.6d 
2.3e 

a  Includes  other  bonds  as  well. 

b  Not  asked. 

c  Nonfarm  real  estate  only. 

d  Secured  debt. 

e  Unsecured  debt. 

Note:  If  mention  was  made  on  a  previous  wave  (a)  of  a  holding  without  giving  its  value  or 
(b)  of  the  total  value  of  a  holding  without  giving  more  details,  e.g.,  number  of  United  States 
savings  bonds,  it  was  considered  a  pickup  on  the  later  wave  when  (a)  value  or  (b)  number  was 
reported. 


posite  was  the  case  for  Study  P2.  This  phenomenon  is  highlighted  in 
Table  56,  which  compares  the  average  size  of  holdings  picked  up  with  the 
average  size  of  holdings  reported  on  the  initial  interview,  by  study  and  by 
holding.  In  the  case  of  Study  PI,  the  average  size  of  the  pickup  is  seen  to 
exceed  considerably  the  average  size  of  the  initial  holding.  As  a  rule,  how- 
ever, the  reverse  is  true  for  Study  P2,  and  for  Study  P3  to  a  lesser  extent. 
The  explanation  for  this  somewhat  contradictory*  result  may  lie  in  the  tim- 
ing of  these  operations  and  in  the  much  higher  representation  of  the  upper- 
income  levels  in  Study  PI.  By  the  time  of  Study  P2,  interviewing  pro- 
cedures had  been  developed  more  fully  and  results  from  Study  PI  could 
be  utilized  as  a  basis  for  special  interviewer  training  in  areas  in  which 
mistakes  or  omissions  appeared  to  be  frequent.  In  addition,  because  of 
the  much  higher  income  level  of  many  of  the  sample  members  in  Study 
PI,  more  time  was  required  to  convince  them  of  the  authenticity  of  the 
study  and  of  the  need  for  full  cooperation.  As  a  result,  a  number  of 
wealthier  sample  members  did  not  begin  to  provide  complete  information 
on  their  holdings  until  the  second  or  third  wave.  By  contrast,  in  the  later 
studies  much  more  stress  was  placed  on  complete  reporting  in  the  very 
first  interview,  so  that  holdings  not  reported  until  later  were  more  likely 
to  represent  inadvertent  omissions  rather  than  deliberate  withholding. 

Although  considerable  variation  occurred  in  the  pickup  of  different 
holdings  from  one  study  to  another,  there  is  some  tendency  for  pickup  to 
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Table  56.  Average  Dollar  Size  of  Pickup  Holding  and  of 
Originally  Reported  Holding,  Studies  PI  to  P3 


Holding 

Study  PI 

Study  P2 

Study  P3 

Initial 

Pickup 

Initial 

Pickup 

Initial 

Pickup 

Checking  accounts 

Savings  accounts 

United  States  government 

bonds 

Stocks  and  other  bonds.  . .  . 

Life  insurance 

Pension  plan  equity 

Loans  to  others 

Real  estate 

Mortgages 

Personal  loans 

$  1,563 
1,161 

304 

4,848 

8,474 

10,769 

2,753 

31,229 

12,437 

1,611 

$  4,265 
8,283 

78 
15,896 
17,278 
28,800 
14,233 
73,267 
10,313 
2,763 

$  1,352 
2,353 

139 
2,968 
4,251 
4,320 
2,873 
16,549 
7,677 
3,503 

$  1,082 
1,297 

99 
1,097 
3,780 
100 
1,405 
16,691 
8,815 
1,186 

$  1,021 
1,061 

456a 
1,246 
2,958 

b 

1,970 

10,201° 

8,109d 

l,469e 

$    176 
1,687 

l,231a 
447 
2,437 

b 

3,000 
440c 

678d 
626e 

a  Includes  other  bonds  as  well. 

b  Not  asked. 

c  Nonfarm  real  estate  only. 

d  Secured  debt. 

e  Unsecured  debt. 


be  highest  for  loans  to  others,  for  personal  loans,  and  for  stocks  and  bonds 
other  than  United  States  Government  savings  bonds.  Even  for  other 
holdings,  pickup  tends  to  represent  anywhere  from  10  to  30  percent  of 
the  total  number  or  value  of  holdings.  Only  in  the  case  of  Study  P3  did 
pickup  amount  to  less  than  10  percent  for  most  holdings. 

Further  investigation  reveals  that  pickup  is  a  characteristic  of  less  than 
half  of  the  sample  members.  In  this  respect,  the  two  urban  studies  pro- 
vide similar  results,  as  is  apparent  from  the  following  tabulations: 

*r     ,  •  Percentage  of  savings  units 

of  pickups  Study  PI       Study  P2 

0 55%  55% 

1 20  15 

2 .  .  : 8  11 

3 8  5 

4 2  3 

5  or  more 6  11 

Total 99%  100% 

Base  (number  of  savings  units) 226  312 

In  each  study,  pickup  was  characteristic  of  45  percent  of  the  panel 
members.  For  most  of  the  savings  units  in  which  pickup  occurred,  two 
or  more  additional  holdings  were  obtained.  In  some  instances,  however, 
five  or  more  additional  holdings  were  obtained,  a  phenomenon  much  more 
frequent  in  Study  P2  than  in  Study  PI. 

Various  cross-tabulations  suggest  that  pickup  of  one  holding  tends  to 
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be  correlated  with  pickup  of  other  holdings.  For  example,  the  following 
tabulation  illustrates  the  extent  to  which  pickup  of  savings  accounts  was 
related  to  pickup  of  life  insurance  in  Study  PI. 

_  . ,                                                         Savings  accounts 
Life  ° 

insurance  Yes  No  Total 

Yes 14  21             35 

No _1£  173  191 

Total 32  194  226 

Response  Accuracy  Over  Time 

Having  examined  some  of  the  overall  effects  of  a  consumer  financial 
panel  operation  on  the  sample,  we  now  turn  to  the  effect  of  the  panel 
on  response  and  nonresponse  errors  in  the  reporting  of  financial  data. 
Such  effects  can  only  be  measured  for  holdings  serving  as  the  basis  for 
primary  validation  in  the  panel  studies,  which  restricts  the  analysis  to 
debt  on  Study  P3,  time  deposits  on  Studies  P2  and  P3,  and  demand  de- 
posits on  Study  P3. 

Farm  Debt 

The  data  requested  from  the  panel  members  of  Study  P3  were  af- 
fected, it  will  be  recalled,  by  two  experimental  manipulations.  First,  half 
of  the  panel  was  interviewed  every  three  months  and  the  other  half  was 
interviewed  every  six  months.  Second,  half  of  the  three-month  panel 
members  and  half  of  the  six-month  panel  members  were  requested  to 
report  holdings  on  the  first  two  waves,  changes  on  the  next  two,  and  hold- 
ings on  the  last  wave;  the  other  half  was  asked  for  change  on  the  first  two 
waves,  holdings  on  the  third,  change  on  the  fourth,  and  holdings  again 
on  the  last  wave.  For  future  reference,  it  is  useful  to  represent  the  fac- 
torial design  diagrammatically  and  to  label  each  group,  as  follows: 

Group 

Wave  A  B  C  D 

1 Holdings  Holdings  Change  Change 

2 Holdings  Change 

3 Change  Change  Holdings  Holdings 

4 Change  Change 

5 Holdings  Holdings  Holdings  Holdings 

As  a  result  of  this  design,  the  following  questions  relating  to  panel 
effects  on  reporting  can  be  investigated,  based  on  comparisons  of  the 
experimental  groups : 

( 1 )  How  do  response  and  nonresponse  errors  in  reports  of  (debt)  hold- 
ings vary  over  time? 
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(2)  Is  a  similar  pattern  evident  among  errors  in  reporting  changes  in 
debt? 

(3)  Are  response  and  nonresponse  errors  fewer  and  smaller  in  reports 
of  holdings  in  interviewing  panel  members  every  three  months  or  every 
six  months?  What  about  reports  of  changes? 

(4)  Do  panel  members  tend  to  report  more  accurately  if  holdings  are 
requested  in  the  very  first  interview  or  if  a  "softer"  approach  is  used  and 
only  change  figures  are  sought  at  the  outset? 

(5)  Do  reports  of  change  improve  after  panel  members  have  been 
asked  for  holdings? 

(6)  Are  changes  computed  from  holdings  data  reported  at  two  points 
of  time  more  accurate  than  changes  reported  directly? 

These  questions  serve  as  the  framework  for  the  analysis  that  follows, 
being  taken  up  first  with  regard  to  number  of  holdings  and  then  with 
regard  to  dollar  amounts. 

Holdings  Reported 

The  extent  to  which  validated  debts  were  reported  on  the  various 
waves  is  shown  in  Table  57,  for  each  of  the  four  experimental  groups 
separately.  The  data  relate  to  individual  debts  rather  than  to  individual 
savings  units.  This  is  because  a  few  savings  units  had  more  than  one 
validated  debt,  which  makes  it  more  convenient  to  carry  out  the  analysis 
on  the  latter  basis.123 

The  table  shows  that  the  number  of  validated  debts  in  each  of  the 
four  subsamples  declined  markedly  throughout  the  course  of  the  panel 
operation.  To  some  extent,  this  was  because  a  certain  number  of  debts 
were  paid  off.124  Deaths  and  moves  out  of  the  area  accounted  for  an 
additional  part  of  this  decline.  Most  important,  however,  was  the  tend- 
ency for  people  to  drop  out  of  the  panel  on  later  waves,  which  was  most 
pronounced  on  the  second  and  third  waves. 

Other  facets  of  the  panel  operation  appear  to  have  acted  in  the  di- 
rection of  improving  the  reliability  of  debt-reporting.  First,  nonresponse 
fell  off  sharply  on  later  waves,  so  that  more  of  the  total  validated  debt  was 
represented  by  the  respondents.  Second,  the  frequency  of  nonreporting  of 
debt  by  respondents  declined  throughout  the  operation,  particularly  on  the 
change  forms  (Groups  C  and  D)  on  the  second  and  third  waves. 

123  Cursory  examination  suggests  that  the  results  would  not  have  been  altered 
in  either  event. 

124  At  the  same  time,  other  debts  were  being  incurred  and  reported.  However, 
these  debts  do  not  properly  belong  in  this  table  and  have  therefore  been  excluded, 
with  the  exception  of  "new"  debts  with  the  primary  validating  institutions.  These 
debts  are  included  because  such  debts  could  be  construed  as  continuations  of 
previous  debts,  and  they  were  so  treated  by  some  institutions. 
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Table  58.  Effect  of  Panel  Operation  on  Reporting 
of  Validated  Debts 


Category 

A.    Holdings, 
three-months 

B.    Holdings, 
six-months 

Wave  1 

After 
five  waves 

Wave  1 

After 
five  waves 

Reported11 

Not  reported 

Nonrespondent 

73% 

16 

11 

68% 

6 
26 

77% 

13 

10 

69% 

5 
26 

Total 

100% 

100% 

100% 

100% 

Base 

64 

65 

61 

62 

Category 

C.    Change, 
three-months 

D.    Change, 
six-months 

Wave  1 

After 

five  waves 

Wave  1 

After 
five  waves 

Reported* 

Not  reported 

Nonrespondent 

64% 
32 
4 

79% 

3 
18 

64% 
30 
6 

78% 
4 
18 

Total 

100% 

100% 

100% 

100% 

Base  (total  validated  debts) 

69 

70 

67 

68 

a  Includes  refusal  of  amount  and  debts  paid  off. 


The  overall  conclusion  from  Table  57  would  seem  to  be  that  the  panel 
operation  served  to  reduce  the  number  of  debts  being  covered,  but  at  the 
same  time  it  rendered  more  reliable  those  debts  that  were  covered.  This 
point  is  brought  out  more  clearly  in  Table  58,  which  compares  the  distri- 
butions of  validated  debts  on  the  first  wave  and  on  the  last  wave  for  each 
of  the  four  experimental  groups.  To  make  the  data  more  comparable,  the 
figures  have  been  converted  into  percentages,  with  the  base  being  the 
total  validated  debts  of  that  group  at  the  particular  time.  As  is  evident 
from  this  table,  the  number  of  debts  reported  on  the  fifth  wave  was  some- 
what, although  not  very  much,  smaller  than  on  the  first  wave  for  Groups 
A  and  B,  from  whom  holdings  were  requested  on  the  first  two  waves.  By 
contrast,  increases  are  apparent  in  the  proportion  and  number  of  debts 
reported  when  changes  were  requested  initially  (Groups  C  and  D).125 

125  To  some  extent,  this  is  due  to  the  inclusion  on  Wave  5  of  debts  paid  off 

as  "reported"  debts.    However,  the  same  phenomenon  remains  if  paid  off  debts  are 

excluded  completely  from  Table  58.    The  distribution  of  total  debts  reported  for 

these  two  groups  then  becomes: 

Wave  1  After  five  waves 


Group  C 64% 

Group  D 64 


73% 
75 
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Table  59.  Nonreporting,,  Nonresponse,  and  Dropout  Rates 
for  Validated  Debts,  Study  P3 


Statistic 

Subsample 

Wave 

1 

2 

3 

4 

5 

Nonreporting  ratea. . . 
Nonresponse  rateb .  .  . 
Dropout  ratec 

A.  Holdings,  3  months 

B.  Holdings,  6  months 

C.  Change,  3  months 

D.  Change,  6  months 

A.  Holdings,  3  months 

B.  Holdings,  6  months 

C.  Change,  3  months 

D.  Change,  6  months 

A.  Holdings,  3  months 

B.  Holdings,  6  months 

C.  Change,  3  months 

D.  Change,  6  months 

18% 
15 
33 
32 

11 
10 

4 

6 

10% 
14 

8 
10 

11 
4 

12% 
16 
10 
12 

8 

11 

3 

9 

17 

10 

15 

6 

10% 
9 

2 
1 

25 
17 

8% 
7 
4 
5 

0 
8 
0 
4 

26 
18 
19 
13 

a  Ratio  of  nonreported  debts  to  validated  debts  of  respondents. 

b  Ratio  of  nonrespondent  validated  debts  to  total  sample  validated  debts. 

c  Ratio  of  number  of  debts  of  dropouts  to  total  sample  debts. 


Table  58  also  brings  out  the  fact  that  the  proportion  of  debts  reported 
initially  was  higher  among  sample  members  asked  for  holdings  than 
among  those  asked  for  change.  The  principal  reason  for  this  was  the 
much  higher  proportion  of  debts  not  reported  on  the  change  form,  a 
phenomenon  associated  with  the  tendency  not  to  report  holdings  for 
which  there  was  no  change,  noted  in  Chapter  III.  Somewhat  surprisingly, 
however,  by  the  fifth  wave  the  proportion  of  debts  reported  was,  if  any- 
thing, higher  among  those  who  had  been  asked  initially  for  changes  than 
among  those  who  had  been  asked  initially  for  holdings.  A  principal  ex- 
planation appears  to  lie  in  the  higher  dropout  rate  among  those  asked 
for  holdings.  Table  58  also  suggests  that  nonreporting  declined  appreci- 
ably among  all  experimental  groups  as  time  went  on,  the  decline  being 
especially  large  for  those  groups  where  a  change  had  been  requested 
initially  (and  where,  apparently  as  a  result,  nonreporting  was  much  more 
substantial) .  This  point  is  brought  out  more  clearly  in  Table  59,  which 
depicts  the  variations  in  the  nonresponse  rate  and  other  rates  over  time. 

In  addition,  the  nonresponse  rate  was  much  higher  among  those  asked 
for  holdings  of  debt  initially  than  among  those  asked  for  changes  initially, 
a  point  also  brought  out  in  Table  59.  As  has  been  noted  previously,  a 
higher  nonresponse  rate  appears  to  be  characteristic  of  the  holdings  form, 
this  form  seemingly  serving  to  "scare  away"  many  borderline  respondents. 

Overall,  these  tables  suggest  that  so  far  as  the  reporting  of  debt  is 
concerned,  the  change  approach  is  as  effective  as  the  holdings  approach 
—  after  a  few  waves  of  interviews.    Initially,  however,  the  holdings  ap- 
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proach  shows  a  clear  advantage  with  regard  to  the  reporting  of  debt  by 
respondents,  but  this  is  obtained  at  the  expense  of  a  higher  nonresponse 
rate  and  of  a  higher  dropout  rate  among  people  asked  for  holdings,  as  is 
clear  from  Table  59.  With  either  approach,  there  is  a  marked  decline  in 
nonreporting  over  time  as  well  as  in  the  proportion  of  sample  members 
not  interviewed  on  a  particular  wave.126 

Dollar  Amounts 

Analysis  of  errors  in  reports  of  dollar  amounts  is  more  complicated  than 
analysis  of  the  number  of  holdings  because  of  the  experimental  manipula- 
tions. On  some  waves,  part  of  the  sample  was  asked  for  dollar  holdings 
and  the  rest  of  the  sample  for  changes  in  debt.  On  other  waves,  the  situa- 
tion was  reversed,  as  noted  earlier.  In  addition,  sample  members  asked 
for  holdings  on  the  third  wave  were  also  asked  to  report  changes  at  the 
same  time.  Hence,  a  number  of  comparisons  relating  to  the  accuracy  of 
dollar  figures  over  time  can  be  made,  some  relating  to  holdings  and  others 
relating  to  changes.  The  comparisons  that  are  made  relate  to  three  differ- 
ent aspects  of  the  panel  operation:  errors  in  debts  reported  or  not  re- 
ported by  respondents,  errors  due  to  differences  between  the  debts  of  the 
nonrespondents  and  the  debts  of  the  respondents,  and  errors  due  to  pos- 
sible biases  because  of  dropouts. 

Respondents.  Errors  in  amounts  based  on  figures  furnished  by  re- 
spondents who  reported  validated  debt  are  shown  in  Table  60.  Perhaps 
the  most  striking  characteristic  of  this  table  is  the  substantial  differences  in 
accuracy  between  the  reports  of  holdings  and  the  reports  of  change,  dif- 
ferences which  persist  throughout  the  study.  The  average  reported  hold- 
ings of  the  debt  of  the  sample  members  deviate  less  than  15  percent  from 
the  true  figure  (although  the  latter  is  consistently  underestimated).  By 
contrast,  the  average  reported  change  in  debt  differs  from  the  true  figure 
by  magnitudes  ranging  from  50  percent  on  up  (with  one  exception) . 
Moreover,  in  both  instances  no  perceptible  trend  is  evident  toward  greater 
accuracy  over  time.   Thus,  these  results  reinforce  the  findings  of  Chapter 

128  The  reason  for  the  decline  in  the  nonresponse  rate  is  that  all  of  the  non- 
respondents  on  Wave  1,  all  but  one  of  whom  were  refusals,  had  dropped  out  of 
the  panel  by  the  next  wave. 

Conversion  of  nonreporters  to  reporters  was  far  more  frequent  among  those 
asked  initially  for  change  than  among  those  asked  initially  for  holdings.  However, 
the  reason  for  the  decline  in  nonreporting  varied  with  the  particular  form  that  was 
used  first.  In  the  case  of  the  holdings  form,  the  decline  was  attributable  primarily 
to  some  of  these  debts  being  paid  off  or  to  a  sample  member  dropping  out  of  the 
panel.  In  contrast,  with  the  change  form,  the  substantial  initial  nonreporting  was 
reduced  by  more  than  two-thirds  because  many  of  these  debts  were  picked  up  on 
later  waves.  The  great  majority  of  these  pickups  occurred  on  the  following  inter- 
view —  Wave  2  for  Group  C,  and  Wave  3  for  Group  D. 
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Table  60.  Errors  Over  Time  in  Validated  Debts 

of  Respondent  Reporters,  Study  P3 

(percentage  of  deviation  in  average  reported  from 

average  actual  debt) 


Wave 


Holdings  initially 


Three  months 


B. 

Six  months 


Change  initially 


Three  months 


D. 

Six  months 


Holdings  reports 


1 

2 

2.2 
11.4 

21 !  7 

5.4 
19^2 

13^2 
26!3 

3 

11.7 

4 

5 

11.0 

Change  reports 


1 

-109.2a 

75!6 
135.2 

-547^6 

20.0 
-47.4 

53.6 
222. 6d 

976. 6b 

2 

3 

10,666.7° 

4 

a  Computed,  rather  than  reported,  change,  based  on  difference  between  holdings  reported  on 
Wave  1  and  on  Wave  2. 

b  Large  percentage  due  to  low  base:  average  actual  change,  — $33.27;  average  reported 
change,  $291.63. 

c  Large  percentage  due  to  low  base:  average  actual  change,  — $5.07;  average  reported  change, 
—$545.87. 

d  Large  percentage  due  to  low  base:  average  actual  change,  — $40.81;  average  reported 
change,  $50.05. 


Ill  regarding  the  much  greater  accuracy  of  reports  of  holdings  than  of 
reports  of  change. 

It  should  be  stressed,  however,  that  this  greater  accuracy  is  in  terms  of 
percentage  deviations,  brought  about  by  the  very  much  higher  base  of  the 
levels  involved  than  of  the  changes.  This  point  is  illustrated  by  the  errors 
in  the  reports  of  Group  C  on  the  third  wave,  a  time  at  which  both  hold- 
ings and  changes  were  requested  from  the  same  people.  As  is  evident 
from  Table  60,  the  average  percentage  of  error  in  the  reports  of  change 
was  roughly  four  times  that  of  the  reports  of  holdings.  Yet,  when  differ- 
ences in  scale  are  taken  into  account,  the  change  reports  appear  in  a  much 
better  light,  as  shown  by  the  following  data : 


Holdings . 
Change .  . 


Percentage 

Average 

Actual  figure 

of  error 

discrepancy 

$6,860 

13.2% 

$1,007 

143 

53.6 

77 

In  other  words,  the  much  smaller  base  from  which  changes  are  mea- 
sured greatly  magnifies  any  resulting  discrepancies.    Still,  this  is  a  phe- 


188     The  Reliability  of  Consumer  Reports  of  Financial  Assets  and  Debts 

nomenon  that  cannot  be  ignored,  for  percentage  deviations  are  usually 
the  most  important  measure  of  the  accuracy  of  such  estimates.  In  terms 
of  these  percentage  deviations,  the  following  additional  inferences  may 
be  drawn  from  Table  60  regarding  panel  effects  on  the  reporting  of  farm 
debt: 

( 1 )  Reports  of  change,  but  not  of  holdings,  appear  to  be  more  accurate 
when  obtained  every  three  months  than  when  obtained  every  six  months. 
To  be  sure,  the  differences  are  not  always  substantial.  In  the  case  of 
change  reports,  however,  pronounced  differences  are  evident  in  all  cases, 
and  the  differences  are  statistically  significant  on  Wave  3.  Such  greater 
differences  on  change  reports  are  not  surprising,  since  one  would  expect 
the  margin  of  possible  error  to  rise  in  such  cases,  due  to  the  increasing 
difficulty  of  selecting  the  base  figure  as  the  time  span  lengthens. 

( 2 )  Panel  members  from  whom  holdings  are  requested  initially  tend  to 
be  no  more  accurate  in  later  waves  than  those  from  whom  changes  are 
requested  initially. 

(3)  Reports  of  change  are  not  much  more  accurate  after  people  have 
been  asked  for  holdings  (compare  Group  A  versus  Group  C  and  Group  B 
versus  Group  D  on  Waves  3  and  4) . 

(4)  In  the  one  instance  in  which  a  three-month  comparison  was  pos- 
sible (Group  A,  Wave  2),  average  change  in  debt  computed  from  holdings 
reports  at  different  points  in  time  was  no  more  accurate  than  change  re- 
ported directly.  However,  the  true  state  of  affairs  may  have  been  the 
opposite,  because  the  computed  changes  were  dominated  by  two  extreme 
errors  not  present  in  the  reported  changes:  in  one  instance  the  error  was 
$23,000,  and  in  the  other  it  was  $14,500.  If  these  extreme  observations 
are  removed,  the  accuracy  of  reported  change  is  as  high  as  that  on  the 
holdings  forms.127 

(5)  The  predominance  of  positive  signs  in  errors  of  change  reflects 
mixed  tendencies.  In  three  of  these  instances  the  actual  change  was  posi- 
tive, and  in"  all  cases  it  exceeded  the  reported  change  in  absolute  amount. 
In  the  other  four  cases  average  debt  declined,  with  the  absolute  change 
greater  than  reported  change  in  two  instances  but  less  than  reported 
change  in  the  other  two  cases.  For  the  two  negative  errors,  reported  debt 
declined  more  than  actual  debt  in  both  cases.  All  in  all,  reported  absolute 
debt  understated  actual  absolute  debt  almost  half  of  the  time  —  four  times 
out  of  nine. 


127  The  propriety  of  such  an  adjustment  could  be  questioned  on  the  grounds 
that  in  practice  such  extreme  observations  tend  to  arise  anyway.  However,  the 
effect  of  one  or  two  extreme  observations  on  a  large-scale  study  would  clearly  be 
far  less  than  in  the  present  case.  Moreover,  in  these  two  instances  some  basis 
existed  for  questioning  the  comparability  of  the  actual  and  the  computed  change 
figures. 
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When  we  expand  the  coverage  of  Table  60  to  include  respondent  non- 
reporters  as  well  as  respondent  reporters,  essentially  the  same  results  are 
obtained.  As  before,  no  improvement  in  accuracy  is  evident  over  time; 
the  tendency  for  holdings  to  be  understated  is  more  pronounced  than  be- 
fore (as  was  to  be  expected),  and  three-month  reports  seem  to  be  more 
accurate  than  six -month  reports,  especially  on  change.  Also  the  data  sug- 
gest that  change  reports  may  be  more  accurate  after  holdings  data  have 
been  requested.  This  time,  computed  change  unadjusted  appears  more 
accurate  than  reported  change  and,  again,  its  accuracy  approaches  that 
of  the  holdings  data  (the  error  drops  from  57  percent  to  5.4  percent)  if 
the  two  extreme  errors  are  removed. 

Roughly  similar  results  are  also  obtained  when  the  various  compari- 
sons are  carried  out  in  terms  of  average  absolute  discrepancies  rather  than 
average  arithmetical  discrepancies,  so  that  errors  of  opposite  sign  do  not 
cancel.  Again,  reports  of  three-month  panel  members  are  more  accurate 
than  those  of  six-month  panel  members  only  in  the  case  of  change,  and 
there  is  no  tendency  for  those  asked  for  holdings  initially  to  report  changes 
later  more  accurately  than  those  asked  for  change  initially.  The  holdings 
reports  are  still  substantially  more  accurate  than  the  change  reports  and 
no  perceptible  trend  toward  greater  accuracy  over  time  is  indicated. 

Considering  the  fact  that  nonreporting  declined  substantially  over 
time,  particularly  among  those  asked  for  change  initially,  it  is  somewhat 
surprising  to  find  no  improvement  in  accuracy  of  reports  of  the  total  re- 
spondents. The  explanation  seems  to  be  twofold.  First,  no  significant 
increase  in  accuracy  of  reporting  occurred  over  time  among  the  respondent 
reporters.  From  the  start,  those  reporting  holdings  were  about  as  accurate 
as  might  be  expected,  and  the  accuracy  of  the  change  reports  failed  to 
improve,  at  least  in  the  aggregate.  Second,  the  debts  of  the  nonreporters 
averaged  considerably  below  those  of  the  reporters.  Hence,  the  non- 
reporters  were  relatively  much  less  important  in  terms  of  dollars  than  in 
terms  of  frequencies,  as  is  also  evident  from  Chapter  III  (Table  10) .  This 
was  not  always  true  of  changes;  except  that  in  this  case,  the  actual 
changes  of  the  nonreporters  bore  little  relationship  to  the  average  changes 
of  the  reporters,  as  is  illustrated  by  the  following  data  for  Group  C : 

Average  changes  of  Actual  changes  of 
Wave                                      reporters  nonreporters 

1 $     148  $-166 

2 -460  -  81 

3 143  -729 

4 -41  110 

Apparently  the  relatively  few  nonreporters,  combined  with  the  diffi- 
culty of  reporting  change,  precluded  any  stability  in  this  relationship,  so 
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that  at  least  for  this  purpose  it  made  little  difference  whether  nonreporters 
were  picked  up  or  not. 

Nonrespondents.  The  foregoing  provides  the  principal  explanation  for 
the  failure  of  the  debt  reports  to  become  more  accurate  over  time,  since 
Table  61  shows  that  errors  due  to  differences  between  the  debts  of  the 
nonrespondents  and  those  of  the  respondents  are  of  little  consequence  in 
an  overall  sense,  both  for  holdings  and  for  changes.  This  is  not  because 
no  appreciable  differences  exist  between  these  two  sets  of  figures :  columns 
4  and  5  of  the  table  suggest  that  the  debts  of  the  respondents  may  differ 
substantially  from  those  of  the  nonrespondents,  and  that  changes  in 
debt  were  more  often  than  not  in  different  directions  for  the  two  groups. 
Rather,  the  primary  reason  is  the  relatively  small  number  of  nonrespon- 
dents, this  group  constituting  not  much  more  than  one-tenth  of  the  panel 
on  all  waves  after  the  first,  and  in  some  cases  containing  no  members  at 
all.  Hence,  the  effect  of  even  substantial  differences  on  sample  estimates 
was  greatly  reduced,  as  is  evident  from  colunms  6  and  7  of  this  table. 

Moreover,  from  the  point  of  view  of  survey  errors,  the  important  thing 
is  not  the  difference  between  nonrespondent  and  respondent  actual  figures 
but  between  the  nonrespondent  and  respondent  reports,  for  it  is  the  latter 
data  that  comprise  the  sample  estimate.  Comparison  of  columns  3  to  5 
of  Table  61  suggests  that  as  often  as  not  the  errors  in  the  respondent 
reports  help  mitigate  the  errors  in  the  overall  survey  estimate,  because 
these  errors  serve  to  bring  the  reported  figures  closer  to  the  nonrespondent 
holdings  than  are  the  actual  debt  figures  for  the  respondents.  This  is  true 
of  3  out  of  7  such  comparisons  for  the  holdings  reports  and  for  5  out  of  9 
comparisons  for  the  change  reports.  It  is  to  a  large  extent  the  instances 
in  which  this  is  not  true  that  the  nonrespondent  effect  is  substantial,  as 
for  Group  G  on  Wave  1  and  for  Group  D  on  Wave  3.128 

Dropouts.  Finally,  we  may  ask  if  panel  mortality,  small  as  it  was, 
exerted  any  cumulative  effect  in  causing  the  panel  to  become  less  repre- 
sentative over  time  of  the  population  from  which  it  was  selected,  at  least 
with  regard  to  debt.  For  present  purposes,  this  question  reduces  to  inves- 
tigating the  extent  to  which  the  holdings,  and  changes,  of  the  dropouts 
deviated  from  the  corresponding  statistics  for  the  nonrespondents. 

The  relevant  data  are  shown  in  Figure  3,  comparing  the  average 
amount  of  debt  of  the  respondents  and  that  of  the  nonrespondents  by 
wave.  Separate  lines  are  shown  for  the  average  debt  of  panel  members 
dropping  out  at  different  times.    The  dollar  amounts  in  this  figure  relate 

128  The  other  principal  reason  is  a  high  frequency  of  nonrespondents  com- 
bined with  a  substantial  difference  in  the  reported  averages  and  nonrespondent 
holdings,  a  phenomenon  most  likely  to  occur  on  the  first  wave,  as  was  the  case  for 
Groups  A  and  B  on  Wave  1. 
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Figure  3.  Average  Debt  of  Respondents  and  of 
Dropouts,  by  Wave,,  Study  P3 
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to  the  validated  debts  of  all  panel  members  (since  the  relatively  small 
number  of  nonrespondents  does  not  make  it  feasible  to  compare  experi- 
mental groups)  and  for  holdings  rather  than  for  changes  in  debt,  because 
bias  would  seem  most  likely  to  occur  in  the  former  case. 

To  judge  by  Figure  3,  differences  between  the  holdings  of  the  respon- 
dents and  those  of  the  nonrespondents  were  relatively  minor  for  most  of 
the  study,  although  the  differences  tended  to  widen  toward  the  end  of  the 
operation.  For  the  first  four  waves,  the  average  for  one  group  is  within 
10  percent  of  the  average  debt  of  the  other  group.  However,  throughout 
the  study  some  tendency  is  apparent  for  those  dropping  out  to  have  more 
debt  than  those  remaining,  a  tendency  that  becomes  most  pronounced  on 
the  last  wave. 

The  comparison  is  also  good  in  terms  of  change  in  debt.  If  change  is 
estimated  as  computed  change  based  on  the  difference  between  average 
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Table  62.  Allocation  of  Error  in  Average  Debt, 
by  Sample  Categories,  Study  P3 


Group 

(2) 

Per- 
cent- 
age of 
error 

(3) 

Allocation  (percent) 

Wave 

0) 

Non- 
re- 

port- 
ing 

(4) 

Re- 
spon- 
dent 

re- 
ports 

(5) 

Amount 
refused 

or  NA 

(6) 

Cur- 
rent 
non- 
re- 
spon- 
dents 

(7) 

Prior 
drop- 
outs 

(8) 

Total 

(9) 

1 

2 
3 

5 

A.  Three  months 

B.  Six  months 

A.  Three  months 

C.  Three  months 

D.  Six  months 

A.  Three  months 

B.  Six  months 

C.  Three  months 

D.  Six  months 

10.3 
10.7 

23.0 

12.4 
23.9 

36.5 
22.3 
16.4 
19.1 

64.4 
106.8 

28.8 

21.6 
54.1 

13.8 
5.1 
1.2 
5.3 

17.8 
43.1 

34.9 

88.9 
34.2 

35.5 

61.2 

108.5 

39.2 

\.5 

17.8 
-49.9 

25.3 

-20.2 
-8.9 

48.3 
44' 6 

11.0 

9.7 
19.1 

50.7 

-14.6 

-9.7 

10.9 

100.0 
100.0 

100.0 

100.0 
100.0 

100.0 
100.0 
100.0 
100.0 

holdings  on  two  successive  waves,  the  figures  coincide  very  well  until 
Wave  5.  At  that  point  the  high  average  debt  of  the  nonrespondents  pro- 
duces appreciable  differences  in  the  change  in  debt  of  this  group  relative 
to  that  of  the  respondents  for  the  last  two  comparisons. 

Similar  results  are  obtained  if  the  comparison  is  made  in  terms  of 
reported  change.  In  that  case,  however,  the  sample  is  reduced  to  only 
those  interviewed  with  change  forms.  The  number  of  nonrespondents  is 
then  very  small,  so  that  sampling  variability  is  greatly  increased.129  Never- 
theless, in  all  but  one  comparison  (on  Wave  5)  the  changes  were  in  the 
same  direction. 

Allocation  of  error.  The  relative  importance  of  various  sources  of 
error  in  reporting  debt  are  evaluated  in  relationship  to  each  other  in 
Table  62,  using  the  same  basic  allocation  principle  developed  in  previous 
chapters.  To  indicate  the  magnitude  of  the  error  involved,  column  3 
shows  the  relative  error  in  the  overall  estimate  of  the  average  debt  of  the 
particular  group  at  that  time. 

This  table  brings  out  the  gradual  shift  in  the  principal  cause  of  error 
from  nonreporting  on  the  first  wave  to  errors  in  the  reports  of  the  respon- 
dents and  to  errors  due  to  dropouts  on  the  later  waves.  From  the  previous 
results,  it  is  clear  that  this  shift  is  attributable  not  so  much  to  fluctuations 
in  the  average  amounts  involved,  but  rather  to  shifts  in  the  proportion  of 

129  This  is  especially  true  because  half  of  the  sample  was  not  interviewed  on 
Waves  2  and  4,  which  affects  the  variability  of  change  estimates  on  all  waves. 
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sample  members  in  each  group  at  a  particular  time.  Thus,  on  the  first 
wave  nonreporting  is  at  a  peak  (both  absolutely  and  relatively) ,  whereas 
nonrespondents  are  at  a  minimum  and  dropouts  are  nonexistent.  On  later 
waves,  the  frequency  of  nonreporting  declines  and  the  proportion  of 
sample  members  who  have  dropped  out  rises  gradually.  While  this  shift 
is  taking  place,  the  relative  influence  of  reporting  error  tends  to  increase, 
as  some  nonreporters  shift  into  the  reporter  category. 

The  table  also  documents  the  tendency  —  apparent  also  from  Figure 
3  —  for  the  sample  averages  to  understate  consistently  the  true  average 
debt.  This  is  primarily  due  to  the  nonreporters  and  to  consistent  under- 
statements by  the  reporters. 

Time  Deposits,  Study  P2 

Because  of  differences  in  the  experimental  design  and  in  the  type  of 
information  obtained,  it  seems  best  to  consider  separately  the  panel  effects 
on  the  reporting  of  savings  accounts  for  the  urban  panel,  Study  P2,  and 
for  the  farm  panel,  Study  P3.  The  former  will  be  considered  first  and 
then  the  latter. 

It  will  be  recalled  (pp.  75-88)  that  four  experiments  were  incorpo- 
rated in  the  first  wave  of  interviews  in  this  study,  relating  to  (1)  the 
type  of  advance  letter,  (2)  the  use  of  questionnaire  forms  in  the  interview, 
(3)  the  type  of  data  requested  (holdings  versus  change),  and  (4)  the 
extent  of  coverage  of  assets  and  debts.  For  the  present  purposes,  only 
the  distinction  by  type  of  data  requested  is  of  primary  relevance,  since  the 
results  on  Wave  1  for  the  other  three  experiments  (pp.  114-16)  do  not  in- 
dicate that  any  of  those  variables  had  any  significant  effect  on  reporting 
of  accounts.  Since  these  experiments  were  of  a  one-time  nature,  panel 
effects  due  to  these  variables  were  hardly  likely,  as  was  verified  by  casual 
examination.  Hence,  for  studying  panel  effects,  the  data-collection  pattern 
over  time  divides  the  sample  in  two,  as  follows : 

Wave  Group  A  Group  B 

1 Holdings  Change 

2 Holdings  Change 

3 Holdings130  Holdings 

4 Holdings  and  change  Holdings  and  change 

5 Holdings  Holdings 

Holdings  Reported 

The  substantial  panel  mortality  on  this  study,  first  noted  on  p.  162,  is 
brought  out  once  again  in  Table  63.    With  reference  to  time  deposits, 

130  On  Wave  3,  members  of  Group  3  were  asked  for  holdings  of  demand  and 
time  deposits  and  of  debt  —  assets  and  debts  with  fluctuating  balances  —  and  for 
change  on  other  assets. 
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Table  63.  Distribution  of  Validated  Time  Deposits, 
by  Sample  Response  Status.,  Study  P2 


Response  status 

Wave 

After 

five 

waves 

1 

2 

3 

4a 

5 

A.  Holdings  first 

Total  sample 

Respondents 

Refused  amount,  not  asked. . 

Nonreporters 

Reporters 

Nonrespondents 

Noncontactsb 

Refusals 

Prior  dropouts0 


240 

243 

245 

246 

246 

169 

153 

150 

141 

139 

8 

9 

7 

21 

8 

56 

40 

32 

31 

28 

105 

104 

111 

91 

103 

71 

31 

11 

17 

6 

22 

9 

8 

3 

4 

49 

22 

3 

14 

2 

59 

84 

86 

101 

246 

139 

8 

28 

103 

107 

26 

81 


B.   Change  first 


Total  sample 

Respondents 

Refused  amount,  not  answered 

Nonreporters 

Reporters 

Nonrespondents 

Noncontacts0 

Refusals 

Prior  dropoutsd 


222 

224 

227 

227 

227 

176 

149 

135 

128 

124 

8 

5 

14 

16 

9 

71 

47 

28 

21 

15 

97 

97 

93 

91 

100 

46 

29 

22 

29 

1 

10 

6 

10 

3 

2 

34 

23 

12 

26 

5 

2 

46 

70 

70 

96 

227 

124 

9 

15 

100 

103 

25 

78 


a  Roughly  half  of  the  sample  was  interviewed  by  mail  on  this  wave. 

b  Includes  accounts  on   12  falsified  questionnaires  on  Wave  1  and  four  falsified  questionnaires 
on  Wave  2. 

c  Includes  two  falsified  questionnaires  on  Wave  1. 
d  Includes  ineligibles. 


refusal  to  grant  an  interview  was  by  far  the  most  important  cause  of  lost 
validated  accounts,  accounting  after  five  waves  for  three-fourths  of  all 
such  accounts  and  for  one-third  of  all  validated  accounts.  It  is  interesting 
to  note  that  although  more  refusals  were  encountered  initially  on  the  hold- 
ings form  than  on  the  change  form,  the  number  of  such  refusals  diminished 
much  less  rapidly  for  the  latter  group.  This  was  particularly  true  of  the 
third  and  fourth  waves,  when  efforts  were  made  for  the  first  time  to  ob- 
tain holdings  information  from  this  group.  As  a  result,  by  the  end  of 
the  operation,  the  rate  of  refusal  was  higher  among  those  asked  initially 
for  change  than  among  those  asked  initially  for  holdings. 

Among  those  who  remained  in  the  panel,  much  the  same  phenomena 
are  apparent  for  time  deposits  as  for  debt  on  Study  P3.  Nonreporting  of 
time  deposits  dropped  through  most  of  the  study,  particularly  on  the  sec- 
ond and  third  waves.  At  the  same  time,  the  rate  of  nonreporting  dropped 
more  sharply  among  those  asked  initially  for  change  —  especially  so  on 
Wave  3  when  balances  were  requested  of  the  members  of  this  group  — 
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with  the  result  that  by  the  end  of  the  field  operation  nonreporting  for  this 
group  was  lower  than  for  those  asked  initially  for  holdings,  as  shown  by 

the  following  percentages : 

Nonreporters  as  percentage  of  all  respondents 

Wave  Holdings  first  Change  first  Total  sample 

1 23%  32%  27% 

2 16  21  19 

3 13  12  13 

4 13  9  11 

5 11  7  9 

This  phenomenon  offset  the  loss  of  sample  members  during  the  course 
of  the  panel,  so  that  the  coverage  of  validated  accounts  actually  increased 
despite  the  substantial  mortality  (almost  45  percent) . 

Also,  similar  to  the  results  for  debt  on  Study  P3  are  the  rapid  declines 
in  the  nonresponse  rate  (nonrespondent  accounts/total  sample  accounts) 
and  in  the  dropout  rate  (dropout  accounts/total  sample  accounts  on  pre- 
vious interview)  .131 

A  similar  result  is  also  obtained  for  the  nature  of  the  shift  of  panel 
members  from  one  form  of  response  to  another,  based  on  relating  a  sample 
member's  status  on  Wave  1  to  his  status  on  Wave  5.  The  decline  in  non- 
reporters  among  those  from  whom  changes  were  requested  originally  re- 
flected primarily  improved  accuracy  of  reporting,  whereas  among  those 
asked  first  for  holdings  the  decline  resulted  principally  from  dropouts.  In 
other  words,  the  latter  were  basically  less  cooperative  from  the  beginning 
and  hence  more  likely  to  refuse  at  a  later  stage. 

Dollar  Amounts 

Respondents.  The  pattern  of  errors  in  dollar  amounts  for  time  deposits 
is  in  many  ways  similar  to  that  for  debt,  as  is  evident  from  a  comparison 
of  Table  64  with  Table  60.  The  average  error  in  balances  reported  by 
respondents  is  virtually  negligible  from  the  start  and,  not  surprisingly, 
registers  no  improvement  over  time.  However,  the  sample  estimate  of  the 
average  balance  of  all  respondents  is  much  more  in  error,  understating 
the  true  figure  on  the  first  two  waves  by  almost  40  percent.  This  under- 
statement is  compounded  primarily  by  two  factors:  the  much  higher  bal- 
ances of  nonreporters  compared  with  reporters,  and  the  roughly  equally 
higher  balances  of  those  refusing  to  give  figures  (whose  balances  were  then 

131  The  only  exception,  a  rise  in  dropouts  after  the  fourth  wave,  was  the  result 
of  a  decision  not  to  interview  further  25  savings  units  which  had  been  consistently 
uncooperative  all  along  and  had  provided  virtually  no  financial  data,  although 
several  had  reported  the  validated  account. 
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Table  64.   Percentage  of  Errors  Over  Time  in  Validated 
Accounts,  Study  P2 


Wave 


Respondent  reporters 


A. 

Holdings  first 


B. 

Change  first 


All  respondents 


Holdings  first 


B. 

Change  first 


Holdings  reports 


1 

2 

3 

4 

2.3 

-1.6 

-6.9 

-5.3 

-11.5 

8.2 

10.0 

-1.1 

39.5 
39.7 
27.1 
17.7 
21.3 

19.8 

37.1 

5 

9.4 

Change  reports 

1 

2 

3 

4 

121  ^8 

109.1 
40,762.5a 

121  ^3 

162'7 

96.8 
226. 6b 

106^6 

a  Large  percentage  error  due  to  low  base;  actual  average  change  was  — $.08,  reported 
average  change,  $32.53. 

■>  Large  percentage  error  due  to  low  base;  actual  average  change  was  — $17.30,  reported 
average  change,  $21.91. 


estimated  as  equal  to  the  average  balance  of  the  reporters) .  The  following 
data  illustrate  the  magnitude  of  these  factors  for  Group  A : 

Average  actual  balance 

Wave                                        Balance  given  Balance  refused  Nonreporters 

1 $2,446                 $3,102  $2,949 

2 1,944                   3,106  3,078 

3 1,938                  4,162  3,141 

4 2,062                   1,650  2,439 

5 1,985                   2,747  2,920 

Only  on  the  fourth  wave  is  no  substantial  underestimation  bias  present 
due  to  the  nonreporters  or  to  the  balance  refusals.  In  addition,  as  was 
noted  in  the  preceding  section,  nonreporting  was  much  more  frequent  on 
the  first  two  waves  than  it  was  later,  which  served  to  depress  the  overall 
sample  estimate  still  further.  The  decline  in  nonreporting  after  the  second 
wave  was  primarily  responsible  for  the  pronounced  improvement  in  the 
sample  estimates  of  all  respondent  balances  on  later  waves,  for  both 
groups.  In  this  respect,  the  panel  effect  was  beneficial. 

For  the  change  reports,  the  results  on  time  deposits  also  parallel  those 
on  debt.  The  average  errors  are  substantial  —  far  above  the  errors  in  the 
reports  of  balances.   Yet,  as  before,  the  dollar  amounts  involved  are  very 
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small,  and  in  these  terms  the  discrepancies  are  not  large.  Wave  4  is  a  case 
in  point,  as  is  shown  by  the  following  figures  relating  to  all  respondents: 

Holdings  Change 

Discrepancy  Discrepancy 

Group                             Amount       {actual-reported)  Amount  {actual-reported) 

A $2,083               $368  $-7  $-11 

B 2,432                908                      180  191 

The  relatively  small  average  change  figures  may  also  supply  much  of 
the  explanation  for  the  failure  of  the  actual  and  reported  figures  to  move 
in  the  same  direction.  Using  plus  to  represent  an  increase  and  minus  to 
represent  a  decline,  the  picture  for  reporters  —  and  essentially  the  same 
for  all  respondents  —  was  as  follows : 

Group  Wave  Actual  change  Reported  change 

A 4  -  + 

B 1  + 

2  -  + 

4  +  - 

In  all  four  instances,  the  reported  and  actual  directions  of  change  did  not 
coincide.  However,  in  all  instances  the  actual  average  changes  were  very 
small,  the  figures  being  $  —  25,  $56,  $—0.08,  and  $68,  respectively.  With 
sample  sizes  of  less  than  100  in  each  case,  the  observed  differences  are 
within  the  range  of  sampling  variation. 

Contrary  to  the  situation  with  holdings,  no  tendency  is  apparent  in 
Table  64  for  the  accuracy  of  the  change  reports  to  improve  with  time, 
nor  do  reports  of  change  appear  to  be  any  more  accurate  after  panel 
members  had  been  reporting  holdings  for  some  time  (compare  the  errors 
in  the  change  reports  for  Groups  A  and  B  on  Wave  4).  On  the  other 
hand,  for  the  two  instances  where  accuracy  of  computed  change  ( the  dif- 
ference between  successive  holdings  reports)  could  be  compared  with  the 
accuracy  of  reported  change,  on  Wave  4  for  Groups  A  and  B,  the  former 
approach  appears  to  be  clearly  superior,  as  is  evident  from  the  following 

data  for  reporters : 

Computed  change  „  - 
£__ _^ percentage  of 

Actual  Estimated  Percentage  of  error  in  reported 

Group                                           amount  amount  error                   change 

A $     134  $         0  100.0%             116.1% 

B -486  -483  -.6                 121.3 

Both  (weighted) -166  -248  -49.4                 121.7 

Two  different  "actual"  changes  are  involved  in  this  comparison  be- 
cause some  panel  members  on  Wave  4  reported  change  but  not  holdings, 
and  a  few  vice  versa. 
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Table  65.    Comparison  of  Time  Deposits  of  Respondents 
and  nonrespondents,  by  group  and  by  wave,  study  p2 


Group 

(1) 

Wave 
(2) 

Average  dollar  amount  for 

Percentage  of  error 
in  estimate  for 

Respondent 
reports 

(3) 

Respondent 
actuals 

(4) 

Non- 
respondents 

(5) 

Re- 
spondents 

(6) 

Total 
sample3 

(7) 

Holdings 

A 

1 

2,391 

2,446 

3,924 

2.3 

47.1 

2 

1,975 

1,944 

4,645 

-1.6 

48.8 

3 

2,172 

1,938 

1,419 

-6.9 

24.8 

4 

2,172 

2,062 

3,150 

-5.3 

22.7 

5 

2,213 

1,985 

2,181 

-11.5 

21.3 

B 

3 

2,331 

2,538 

3,997 

8.2 

19.8 

4 

1,848 

2,052 

3,836 

10.0 

37.3 

5 

2,315 

2,289 

8,518 

-1.1 

9.4 

Change 


A 

B 

4 

1 
2 
4 

4 

-5 

33 

-15 

-7 
56 

b 

68 

313 

-86 
-68 
386 

121.8 

109.1 

40762 . 5 

121.3 

87.6 
96.8 

226.6 
106.6 

a  Excluding  prior  dropouts. 
b  Less  than  $.50. 


Nonrespondents.  Differences  are  apparent  between  average  holdings, 
and  changes,  of  the  nonrespondents  and  those  of  the  respondents  (Table 
65 ) .  Balances  of  nonrespondents  averaged  considerably  above  those  of 
the  respondents  and,  perhaps  because  of  the  foregoing,  changes  in  bal- 
ances were  much  more  pronounced.  Since  respondent  reports  tended  to 
understate  the  true  holdings,  and  changes,  of  respondents  anyway  (be- 
cause of  sizeable  nonreporting),  the  result  of  this  upward  bias  in  the 
holdings  of  the  nonrespondents  is  to  increase  the  error  in  the  sample 
estimate  of  the  average  balance  of  all  panel  members.  On  later  waves 
this  tendency  was  mitigated  by  the  sharp  decline  in  the  number  of  non- 
respondents. 

The  errors  in  the  estimates  of  change  are  reduced  as  a  result  of  adding 
the  figures  for  the  nonrespondents.  However,  this  is  the  result  of  an  arith- 
metical quirk  stemming  from  the  increase  in  the  base  for  the  calculation 
of  the  percentage  errors  as  a  result  of  combining  the  actual  changes  for 
the  respondents  with  the  much  higher  figures  for  the  nonrespondents. 

Dropouts.  The  average  account  sizes  of  the  dropouts  were  consistently 
above  those  of  the  respondents  (Figure  4) .  Thus,  on  Wave  1  the  average 
balance  in  the  validated  accounts  of  the  respondents  was  $2,842,  whereas 
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Figure  4.  Average  Size  of  Time  Deposit  Accounts  of 
Respondents  and  of  Dropouts,  by  Wave,  Study  P2 
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that  of  the  dropouts  was  almost  30  percent  larger,  $3,624.  This  margin 
widened  gradually  during  later  waves  of  the  panel  operation.  As  a  result, 
by  the  last  wave,  the  average  balance  of  the  respondents  had  declined  to 
$2,255,  whereas  that  of  the  dropouts  had  risen  to  $3,922.132  Although 
some  discrepancies  are  evident  in  the  wave-to- wave  changes  in  the  average 
balances  of  the  two  groups,  the  overall  changes  are  similar:  holdings  of 
respondents  in  these  accounts  dropped  by  20  percent,  whereas  those  of 
nonrespondents  rose  by  8  percent. 

As  a  result  of  these  diverse  movements,  little  correspondence  is  ap- 
parent between  the  saving  of  respondents  and  the  saving  of  dropouts  in 
these  accounts.  It  might  be  noted  that  the  pattern  of  changes  in  the  hold- 
ings of  the  dropouts  is  influenced  primarily  by  that  of  those  dropping  out 

132  This  decline  does  not  necessarily  suggest  that  the  savings  accounts  of  these 
panel  members  declined  over  time,  for  the  comparison  is  restricted  to  the  original 
validated  accounts.  Excluded  are  all  nonvalidated  accounts  as  well  as  new 
accounts. 
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Table  66.    Percentage  Allocation  of  Error  in  Average  Time 
Deposit  Balances,  by  Source  of  Error  and  by  Wave,  Study  P2 


Allocation 

A.  Holdings  first: 
Wave 

B.  Change  first: 
Wave 

1 

2 

3 

4 

5 

3 

4 

5 

Nonreporting 

48.7 
1.7 

1.6 
48.0 

34.0 
-.7 

2.9 
29.1 
34.7 

32.5 

-7.7 

4.3 

-1.1 

72.0 

24.5 
-3.1 

-3.6 

9.4 

72.8 

28.9 
-7.9 

1.5 

.9 

76.6 

16.5 

7.6 

.5 
21.1 
54.3 

18.0 

5.1 

9.9 
20.4 
46.6 

7  6 

Errors  in  balances 

reported 

-.9 

Errors  in  estimated 
balances  of  those  re- 
porting account  but 
refusing  balance 

Errors  in  estimated 
balance  of  current 
nonrespondents 

Errors  in  estimated 
balances  of  prior 
dropouts 

2.4 
17.0 
73.9 

Total 

100.0 

100.0 

100.0 

100.0 

100.0 

100.0 

100.0 

100.0 

Overall  percentage  of 
error 

47.1 

51.9 

42.8 

40.9 

38.9 

36.9 

49.9 

37.5 

on  the  first  wave.  From  a  statistical  point  of  view  this  is  only  to  be  ex- 
pected, because  the  latter  group  represents  more  than  half  of  all  dropouts. 
Allocation  of  error.  As  was  found  in  the  case  of  farm  debt,  nonreport- 
ing is  seen  to  be  the  principal  source  of  error  at  the  beginning  of  the  opera- 
tion, accounting  for  nearly  half  of  the  total  error  on  the  first  wave  (Table 
66) .  Over  time,  the  relative  importance  of  this  source  tends  to  diminish, 
principally  because  of  the  declining  relative  frequency  of  nonreporters.  In 
contrast,  errors  in  the  estimated  balances  of  nonrespondents  and  of  drop- 
outs increase  in  importance  over  time,  as  would  be  expected.  By  the  last 
two  waves,  these  sources  account  for  75  to  90  percent  of  the  overall  per- 
centage of  error  in  the  average  balance.  This  increase  is  attributable  both 
to  the  rising  proportion  of  the  original  panel  in  these  categories  and  to  the 
growing  extent  to  which  the  reported  balances  understate  those  of  the 
dropouts,  as  is  supported  by  the  following  data : 

Percentage  of  discrepancy 
between  average  balance 

of  nonrespondents  plus  Panel  nonrespondents  and 

dropouts,  and  reported  dropouts  as  percentage  of 

Wave  balance  of  all  respondents  original  sample  size 

1 22%  25% 

2 31  36 

3 40  41 

4 39  44 

5 42  46 
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Compared  with  these  large  sources  of  error,  errors  in  the  balances  of 
accounts  reported  by  respondents  are  virtually  negligible,  as  was  true  also 
of  errors  in  the  use  of  respondent  reports  to  estimate  balances  of  accounts 
for  which  figures  were  refused. 

Time  Deposits,  Study  P3 

The  validation  of  savings  accounts  on  the  farm  panel  followed  the 
same  experimental  design  used  on  farm  debt,  as  described  in  pp.  181-94. 
It  is  perhaps  needless  to  note,  however,  that  different  respondents  were 
involved  in  the  two  validations. 

Holdings  Reported 

The  distribution  of  validated  accounts  over  time  by  sample  member 
status  is  shown  in  Table  67.  As  with  the  other  validation  analyses,  panel 
mortality  was  fairly  high,  about  30  percent  of  the  validated  accounts  being 
owned  by  panel  members  who  had  dropped  out  by  the  last  wave  of  inter- 
views. However,  roughly  half  of  these  dropouts  occurred  on  the  first 
interview,  before  the  sample  members  knew  of  the  panel  aspects,  so  that 
the  panel  feature  accounts  for  a  loss  of  only  about  15  percent  of  the 
original  sample. 

As  was  true  in  previous  instances,  most  of  this  loss  is  represented  by 
refusals.  Noncontacts  were  few,  particularly  since  four  of  the  nine  non- 
contacts  listed  in  the  table  represented  people  who  had  either  moved  out 
of  the  area  or  who  had  given  up  farming  and  were  no  longer  eligible  for 
interview.  Comparing  the  different  experimental  groups,  a  much  higher 
mortality  is  seen  to  occur  among  those  asked  first  for  holdings,  as  shown 
by  the  following  tabulation : 

Refusals  as  a  percentage  of  total  sample 

Wave  A  B  C  D 

1 23%         19%         13%         10% 

'3 20  27  21  10 

5 35  35  25  10 

Since  panel  members  originally  interviewed  with  a  change  form  were 
switched  to  a  holdings  form  on  Wave  3,  the  refusal  rate  for  these  groups 
(C  and  D)  would  be  expected  to  rise  at  that  time.  This  is  seen  to  be  true 
of  Group  C,  but  with  Group  D  the  refusal  rate  was  not  only  initially  very 
low  but  no  additional  refusals  were  encountered  on  later  waves. 

Table  67  also  reaffirms  the  much  higher  incidence  of  nonreporting 
among  those  asked  for  changes  than  among  those  asked  for  holdings.  This 
is  brought  out  more  clearly  in  the  following  tabulation  of  nonreporters  as 
a  percentage  of  total  respondents,  for  each  experimental  group  on  Waves 
1,  3,  and  5. 
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Nonreporters  as  a  percentage  of  respondents 
Wave  A  B  C  D 

1 29%         24%         53%         35% 

3 24  13  39  19 

5 13  12  35  9 

Contrary  to  the  findings  of  Study  P2,  the  nonreporting  rate,  although 
declining  for  all  groups  over  time,  remains  appreciably  lower  for  the  two 
groups  (A  and  B)  asked  originally  for  holdings,  except  for  that  of  Group 
D  on  Wave  5.  It  is  somewhat  surprising  to  note  that  the  largest  decline 
in  nonreporting  occurs  between  Waves  1  and  5  for  the  people  interviewed 
every  six  months.  The  data  in  Table  67  also  support  earlier  findings  that 
nonreporters  among  members  asked  for  holdings  were  most  likely  to  drop 
out  of  the  panel  on  a  later  wave,  whereas  nonreporters  asked  for  change 
tended  to  stay  in  the  sample,  with  an  appreciable  number  reporting  the 
account  on  a  later  interview. 

Dollar  Amounts 

Respondents.  Table  68  shows  that  the  balances  reported  are  reason- 
ably accurate;  only  for  Group  A  on  the  first  wave  is  there  any  substantial 
error.133  Nonreporting  and  the  relatively  large  size  of  accounts  for  which 
the  balance  was  refused  leads  to  consistent  understatement  of  the  average 
balance  of  all  respondents,  but  the  accuracy  of  reports  clearly  improves 
over  time,  as  it  did  in  Study  P2.  The  following  tabulation,  which  com- 
bines Groups  A  and  B  to  reduce  sampling  variation,  shows  that  the  actual 
balance  in  the  accounts  that  were  reported  averages  considerably  below 
the  balances  in  accounts  not  reported  or  in  accounts  for  which  the  balance 

was  refused : 

Average  actual  balance 

Groups  Wave 

A  and  B 1 

5 
CandD 3 

5 

This  tabulation  also  suggests  that  the  error  reduction  over  time  in  the 
balances  of  all  respondents  is  due  primarily  to  the  decline  in  nonreporters 
rather  than  to  any  closer  correspondence  between  the  size  of  accounts 
reported  and  the  size  of  accounts  not  reported. 

As  before,  reports  of  change  are  considerably  more  in  error  percentage- 
wise  than   reports   of  holdings.     (The   strikingly   high   accuracy   of   the 

133  The  large  overstatement  on  Wave  5  is  attributable  almost  entirely  to  one 
large  discrepancy  in  which  the  institution  reported  a  balance  of  roughly  $100  and 
the  respondent  reported  a  balance  of  $12,000. 


Given 

Refused 

Not  reported 

$1,329 

$6,588 

$2,327 

1,629 

6,515 

3,545 

1,657 

5,032 

1,123 

1,942 

4,260 

1,677 
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Table  68.   Percentage  of  Errors  Over  Time  in  Validated  Time 
Accounts,  Study  P3 


Wave 


Holdings  initially 


Three  months 


B. 

Six  months 


Change  initially 


Three  months 


D. 

Six  months 


Respondent  i 

eporters:    holdings 

1 

2 

3 

4 

47.3 
9.3 

-88  ioa 

4.1 
13^9 

-6^6 
-.2 

-4.9 

5 

-3.1 

Respondent 

reporters:   change 

1 

2 

3 

4 

125.6 
95.3 

-70A 

-10.7 
237.7 

104^7 

150. 6b 

All  respondents:    holdings 


1 

65.2 

57.8 

2 

28.0 

3 

29.2 

15.0 

4 

5 

-29.9° 

38.5 

21.3 

8.6 

All  respondents:    change 


1 

-.5 

218. 4d 

2 

106.1 

3 

129.3 

-70.3 

4 

94.4 

104.7 

a  If  one  extreme  discrepancy,  of  almost  — $12,000,  were  removed,  the  error  would  be 
—  9.5  percent. 

b  Large  percentage  due  to  low  base:  average  actual  change,  — $10.09;  average  reported 
change,  —$25.29. 

c  If  one  extreme  discrepancy,  of  almost  — $12,000,  were  removed,  the  error  would  be  23.2 
percent. 

d  Large  percentage  due  to  low  base:  average  actual  change,  $13.97;  average  reported 
change,  — $16.54. 


average  change  report  on  the  first  wave  for  Group  C  appears  to  be  some- 
what of  a  fluke,  resulting  from  fairly  close  correspondence  between  the 
reported  and  actual  changes  of  the  reporters  and  from  the  minimization 
of  the  change  in  the  accounts  of  the  nonreporters  because  of  the  very 
small  amounts  of  change  involved. ) 

The  errors  in  the  average  change  in  balances  tends  to  exceed  by  far 
the  errors  in  average  balances  themselves.  Contrary  to  the  findings  of 
Study  P2,  many  of  the  average  changes  obtained  in  this  study  were  not 
small,  some  amounting  to  several  hundred  dollars.  Nevertheless,  as  shown 
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Table  69.  Average  Balances  and  Changes  in  Balances  in  Time 
Deposits  of  Respondents  and  of  Nonrespondents,  Study  P3 


Group 

Wave 
(2) 

Average  dollar  amount  for 

Percentage 
of  error  in 
estimate  for 

(1) 

Respondent 
reports 

(3) 

Respondent 
actuals 

(4) 

Non- 
respondents 

(5) 

Respon- 
dents 

(6) 

Total 
sample 

(7) 

Holdings 

A  and  B 

G  and  D 

1 

5 

3 

5 

743 
1,963 

1,285 
1,617 

1,919 
1,629 

1,665 
1,947 

2,865 
5,195 

1,000 
8,555 

61.3 
-20.1 

22.8 
16.9 

64.7 

5.5 

20.4 
37.1 

Change 

A  and  B 

G  and  D 

3 

1 

-83 

42 

600 
59 

24 

25 

113.8 
28.8 

113.8 
10.4 

by  the  following  tabulation  for  reporters,  the  direction  of  change  was 
frequently  incorrect : 

Group 

A 


D. 


Wave 

Actual  change 

Reported  change 

3 

$1,668 

$-417 

4 

118 

6 

3 

129 

220 

1 

227 

251 

2 

-46 

64 

4 

289 

-14 

1 

-10 

-25 

Once  again,  computed  change  makes  a  better  showing  than  reported 
change,  although  only  one  comparison  is  possible.  The  result  of  this 
comparison,  based  on  computed  change  for  Group  A  between  Wave  1 
and  Wave  2  versus  reported  change  for  the  same  period  for  Group  C,  is 
as  follows : 

Computed  average  change,  Group  A: 

Actual $1,062 

Estimated $1 ,  716 

Percentage  of  error 61.6% 

Reported  average  change,  Group  C: 

Percentage  of  error 237.7% 

Nonrespondents.  Holdings  of  the  nonrespondents  were  again  higher 
than  those  of  the  respondents,  to  judge  by  Table  69.  Changes  in  the 
holdings  of  the  nonrespondents  were  appreciably  less,  in  absolute  terms, 
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than  changes  in  the  holdings  of  respondents,  contrary  to  findings  in  Study 
P2.  However,  the  base  for  many  of  these  averages  is  very  small  —  gen- 
erally between  three  and  five  observations  for  the  nonrespondents  —  and 
therefore  not  much  weight  can  be  placed  on  these  comparisons. 

Dropouts.  In  contrast  to  Study  P2,  Study  P3  showed  little  correspond- 
ence between  the  holdings  of  the  panel  members  and  those  of  the 
dropouts  over  time.  The  holdings  of  the  nonrespondents  and  of  the  re- 
spondents were  roughly  similar  on  Wave  1,  but  the  pattern  of  change  dif- 
fered thereafter.  Respondent  average  balances  rose  to  Wave  3  and  de- 
clined from  Wave  3  to  Wave  5,  whereas  dropout  balances,  dominated  by 
those  of  Wave  1  nonrespondents,  followed  an  opposite  course.  However, 
taking  the  period  as  a  whole,  the  balances  of  both  groups  rose  noticeably, 
although  much  more  on  the  part  of  the  balances  of  nonrespondents. 

Allocation  of  error.  Nonreporting  was  the  principal  initial  source  of 
error  but  declined  in  importance  on  later  waves  as  the  number  of  non- 
reporters  declined  and  the  number  of  dropouts  increased.  Errors  in  re- 
spondent reports  of  the  balances  provided  another  major  source  of  error 
for  Group  A;  however,  this  source  of  error  was  minor  for  the  other  three 
groups.  Especially  significant  is  the  fact  that,  as  in  Study  P2,  the  accuracy 
of  the  panel  data  once  more  is  seen  to  increase  over  time. 

Demand  Deposits 

The  initial  sample  for  validation  of  demand  deposits  in  Study  P3  was 
smaller  than  that  for  the  other  assets.  Also,  due  to  difficulties  in  obtaining 
balances  on  the  initial  wave,  the  sample  size  for  analysis  of  errors  in  the 
dollar  figures  on  Wave  1  was  considerably  smaller  than  the  sample  size 
serving  as  the  basis  for  analysis  for  reporting  and  nonreporting  of  ac- 
counts. It  might  also  be  noted  that  after  the  first  wave,  when  the  greater 
accuracy  of  the  reporting  of  these  accounts  became  apparent,  more  em- 
phasis was  placed  on  asking  for  holdings.  As  a  result,  changes  were  sought 
on  later  waves  only  on  Wave  4. 

Holdings  Reported 

The  time  pattern  of  the  distribution  of  77  validated  demand  deposits, 
shown  in  Table  70,  presents  a  more  favorable  picture  than  either  time 
deposits  or  debt  for  the  same  study.  Panel  mortality  on  demand  deposits 
was  slightly  lower  than  that  for  the  other  holdings.  About  15  percent  of 
those  asked  for  holdings  on  the  first  wave  dropped  out,  as  was  true  of  a 
much  smaller  proportion  (5  percent)  of  those  asked  for  changes.  By  the 
end  of  the  operation,  panel  mortality  amounted  to  roughly  one-third  of 
those  asked  initially  for  holdings,  but  it  was  only  about  half  as  much  for 
those  asked  initially  for  changes. 
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Table  71.   Percentage  of  Errors  Over  Time  in  Validated 
Deposit  Accounts,,  Study  P3 


Wave 


Holdings  initially 


A. 

Three  months 


B. 

Six  months 


Change  initially 


Three  months 


D. 

Six  months 


Respondent  reporters:    holdings 


1 

5.9 

-65.6 

2 

4.7 

16.8 

3 

14.4 

-.9 

-3.9 

.3 

4 

5 

24.4 

35.0 

16.1 

22.3 

Respondent  reporters:   change 


All  respondents:   change 


1 

2 

3 

4 

131.8 

-25.2 
49.0 

74.7 

All  respon 

dents:    holding 

s 

1 

7.1 

-56.2 

2 

.5 

52.0 

3 

14.4 

-6.9 

58.8 

13.3 

4 

5 

24.4 

35.0 

53.4 

31.9 

1 

136.3 

76.9 

2 

3 

4 

131.8 



56.5 

In  addition,  it  is  significant  to  note  that,  as  before,  most  of  the  drop- 
outs among  those  asked  initially  for  holdings  represented  refusals,  whereas 
nearly  half  of  the  dropouts  from  the  other  groups  were  noncontacts. 

Nonreporting  of  demand  deposits  was  appreciably  less  than  that  for 
the  other  holdings  studied,  even  on  the  first  wave  and  even  for  those 
asked  initially  for  changes.  Of  this  latter  group,  only  three  out  of  36 
accounts  were  not  reported  on  the  first  wave,  the  number  dwindling  to 
one  by  the  last  wave.  For  those  asked  initially  for  holdings,  two  out  of 
33  accounts  were  not  reported,  but  by  the  last  wave  all  of  the  validated 
accounts  of  the  respondents  were  being  reported. 

Dollar  Amounts 

Respondents.  The  magnitude  of  error  in  reports  of  checking  accounts 
seems  to  be  slightly  less  than  for  savings  accounts,  to  judge  from  Table  71. 
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As  with  savings  accounts  and  debt,  reports  of  balances  tend  to  be  consid- 
erably more  accurate  than  reports  of  change,  and  no  trend  toward  greater 
accuracy  in  balances  over  time  is  apparent.  Respondent  reports  of  bal- 
ances tend  to  be  distributed  almost  equally  between  overstatements  and 
understatements,  although  the  average  balance  of  all  respondents  is,  as 
before,  understated.  Some  tendency  can  be  observed  for  accounts  for 
which  the  balance  is  refused  to  be  larger  than  those  that  are  reported,  as 
is  also  true  of  accounts  not  reported  for  Groups  C  and  D. 

Reports  of  change,  although  still  less  accurate  than  reports  of  holdings, 
exhibit  smaller  discrepancies  than  was  true  of  savings  accounts.  Also,  in 
the  present  instance,  the  direction  of  change  is  correct  in  three  cases  out 
of  four,  as  shown  by  the  following  data: 

Groups                             Wave  Actual  change  Reported  change 

A 4  $     412  $-131 

C 1  -133  -167 

C 4  791  403 

D 1  493  125 

Nevertheless,  as  in  previous  instances,  computed  change  appears  to 

be  considerably  more  accurate  than  reported  change,   although  direct 

comparisons  are  not  possible.    Errors  for  computed  change  are  shown 

below: 

Group  A  Group  C 

Wave  1  to  Wave  2  to  Wave  2  to 

Wave  2  Wave  3  Wave  3 

Actual  change $1,151         $-1,093  -362 

Estimated  change 1,283  -1,385  -90 

Percentage  of  error -11.5%  -26.7%  -75.2% 

The  one  case  in  which  computed  change  is  substantially  in  error  occurs 
when  the  actual  change  was  relatively  small.  In  the  other  two  instances, 
the  percentage  of  error  in  the  computed  change  figures  is  below  that  of 
each  of  the  four  errors  for  reported  change  shown  in  Table  71  for  other 
groups. 

Nonrespondents.  The  relatively  few  nonrespondents,  particularly  on 
later  waves,  make  it  difficult  to  judge  whether  or  not  dollar  figures  for  this 
group  are  appreciably  different  from  those  of  the  respondents.  The  data 
suggest,  as  in  the  case  of  time  deposits,  that  the  balances  of  the  nonre- 
spondents are  higher  than  those  of  the  respondents  and  that  this  difference 
tends  to  persist  over  time.  However,  in  no  case  is  the  average  for  the 
nonrespondents  based  on  more  than  six  observations. 

Dropouts.  Very  different  comparisons  between  holdings  of  respond- 
ents and  of  dropouts  are  obtained,  depending  on  whether  the  comparison 
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Table  72.  Allocation  of  Error  in  Average  Demand  Deposit  Balance, 
by  Source  of  Error  and  by  Wave,  Study  P3 


Groups 

(2) 

Per- 
centage 
of  error 

(3) 

Allocation  (percent) 

Wave 

(1) 

Non- 
report- 
ing 

(4) 

Respon- 
dent re- 
porters 

(5) 

Amount 

refused 

or  not 

available 

(6) 

Current 

non- 
respon- 
dents 

(7) 

Prior 
drop- 
outs 

(8) 

Total 

(9) 

1 

2 

3 

5 

A  andB 

A  and  B 
CandD 

A  and  B 
CandD 

A  and  B 
CandD 

-34.5 

-13.4 
48.8 

12.6 
39.2 

27.1 
44.1 

-.7 

95'4 

.4 
70.2 

11.5 

114.4 

32.5 

17.9 

44.0 
-1.7 

81.2 

24.6 

32.0 

-8.1 

-15.8 
6.3 

39.4 

-17.0 
29.0 

-35.1 
28.5 

-5.9 
9.4 

103.5 

-5.2 

106.5 
-3.3 

24.8 
15.1 

99.9 

100.0 
100.0 

100.0 
100.0 

100.1 
100.0 

is  based  on  those  interviewed  every  three  months  or  every  six  months 
(Waves  1,  3,  and  5)  or  only  on  those  interviewed  every  three  months  (for 
which  Waves  2  and  4  are  comparable) .  In  the  former  case,  the  holdings 
of  the  dropouts  are  substantially  above  those  of  the  respondents,  but  the 
two  averages  approach  each  other  over  time,  virtually  coinciding  on  the 
last  wave.  In  the  latter  case,  the  respondent  balances  are  much  higher 
and  no  tendency  toward  a  juncture  is  apparent  (although  only  two  points, 
each  based  on  many  fewer  observations,  are  involved) . 

In  either  case,  the  direction  of  change  in  respondent  balances  coincides 
with  that  of  the  dropouts.  Also,  as  was  found  on  the  other  validation 
studies,  the  time  pattern  of  the  holdings  of  the  dropouts  is  dominated  by 
that  of  the  nonrespondents  on  the  first  wave ;  the  latter  represented  nearly 
half  of  the  total  dropouts  but  —  somewhat  surprisingly  —  less  than  one- 
fifth  of  the  total  dollar  balances  of  the  dropouts  (as  of  Wave  5) . 

Allocation  of  error.  Nonreporting  as  a  source  of  error  on  the  initial 
waves  seems  much  less  because  of  the  relatively  low  incidence  of  this 
phenomenon  (Table  72)  ;  it  appears  for  Groups  C  and  D  only  because 
the  one  nonreported  account  had  a  very  high  balance  on  Waves  1  and  3. 
The  later  influence  of  prior  dropouts  is  also  much  less  in  evidence,  as  a 
result  of  the  closer  correspondence  between  respondent  and  dropout 
holdings  over  time.  Nevertheless,  the  respondent  averages  tend  to  under- 
state the  overall  sample  average  appreciably  on  later  waves,  due  partly  to 
understatement  of  balances  reported  and  of  estimated  sizes  of  balances 
refused.  Again,  however,  observations  are  few,  and  it  is  difficult  to  draw 
generalizations. 


212     The  Reliability  of  Consumer  Reports  of  Financial  Assets  and  Debts 

Overall  Evaluation 

The  results  of  this  chapter  suggest  that  pronounced  differences  occur 
over  time  in  the  composition  of  the  panel,  in  the  cooperativeness  of  the 
panel  members,  and  in  the  reliability  of  the  financial  data  that  are 
obtained.  Essentially,  those  remaining  in  a  panel  operation  are  more  likely 
to  be  younger  families  and  are  also  more  likely  to  be  in  the  middle  of  the 
range  in  terms  of  value  of  home.  There  is  some  indication  that  con- 
tinuing panel  members  possess  somewhat  less  wealth  than  panel  members 
who  drop  out,  but  such  evidence  is  only  circumstantial. 

These  studies  do  provide  clear  evidence  that  cooperativeness  tends  to 
increase  over  time,  principally  in  the  earlier  stages  of  a  panel  operation. 
Moreover,  respondent  cooperation  varies  noticeably  with  the  character- 
istics of  the  respondents,  being  higher  among  those  with  more  education 
and  those  in  laboring  occupations  and  lower  among  older  people  and 
those  in  the  higher  income  groups.  However,  there  is  little  evidence  to 
support  the  hypothesis  that  respondent  cooperation  is  influenced  by  the 
gross  value  of  assets  or  by  the  size  of  savings  account  balances. 

The  principal  cause  of  improved  cooperation  over  time  is  shown  to  be 
differential  panel  mortality.  Those  who  are  less  cooperative  on  the  first 
interview  are  more  likely  to  drop  out  of  the  panel  on  later  interviews.  At 
the  same  time,  rapport  seems  to  increase  among  those  remaining  in  the 
panel  operation. 

Pickup  of  holdings  on  later  interviews  varies  substantially  with  the 
holding  and  with  the  study.  However,  in  most  instances  pickup  amounts 
to  more  than  10  percent  of  total  holdings,  whether  computed  on  the  basis 
of  number  or  on  the  basis  of  dollar  value.  The  fact  that  two  or  more 
pickups  were  obtained  among  25  to  30  percent  of  the  panel  members  in 
these  studies  supports  the  later  findings  that  pickup  for  one  holding  tends 
to  be  correlated  with  pickup  for  other  holdings.  This  suggests  that  once  a 
single  pickup  is  obtained  it  might  well  be  worth  probing  for  additional 
holdings  not  reported. 

Although  each  of  the  validation  analyses  is  based  on  a  relatively  small 
sample,  taken  together  they  enable  a  number  of  inferences  to  be  drawn 
regarding  the  effect  of  a  panel  operation  on  the  accuracy  of  reporting  of 
financial  holdings.  One  such  inference  is  that  despite  appreciable  panel 
mortality  over  time,  improvement  in  the  accuracy  of  the  data  reported  by 
those  remaining  in  the  panel  tends  to  more  than  offset  bias  due  to  panel 
mortality. 

Second,  there  is  some  evidence  that  the  holdings  of  nonrespondents 
exceed  those  of  respondents.  This  bias  tends  to  remain  fairly  constant 
over  time. 


Panel  Effects  213 

Third,  the  principal  causes  of  error  in  the  reporting  of  financial  data 
on  the  initial  waves  of  a  panel  operation  appear  to  be  the  relatively  high 
incidence  of  nonreporters  and  the  higher  average  holdings  of  nonre- 
spondents,  both  acting  to  produce  underestimates.  These  underestimates 
continue  to  be  manifested  in  the  later  stages  of  the  study  as  well,  the 
dwindling  number  of  nonreporters  being  offset  by  an  increasing  number 
of  dropouts  with  relatively  high  holdings. 

Fourth,  the  accuracy  of  reports  of  dollar  holdings  tends  to  be  sub- 
stantially higher  than  the  accuracy  of  reported  changes  in  holdings. 
However,  on  neither  basis  does  overall  accuracy  improve  noticeably  over 
time. 

Fifth,  computed  change,  estimated  as  the  difference  between  two 
successive  reports  of  dollar  balances,  is  substantially  more  accurate  than 
change  figures  obtained  directly.  Where  large  changes  are  involved,  the 
accuracy  of  computed  change  reports  compares  favorably  with  that  of 
reported  dollar  balances. 

Sixth,  nonreporting  when  people  are  asked  for  change  is  appreciably 
greater  than  nonreporting  when  people  are  asked  for  dollar  balances  but, 
as  has  been  noted  in  previous  chapters,  refusals  tend  to  be  much  higher 
in  the  latter  case.  Nonreporting  for  both  groups  drops  over  time,  but 
much  more  so  for  those  asked  initially  for  change.  Interestingly  enough, 
a  tendency  is  evident  for  nonreporters  of  dollar  balances  to  drop  out  of 
the  panel  at  a  later  stage  and  for  nonreporters  of  dollar  changes  to  remain 
in  the  panel  and  report  the  holding  on  a  later  interview. 

If  one  thing  is  clear  from  these  results,  it  is  that  a  consumer  panel 
operation  tends  to  improve  the  reliability  with  which  financial  data  are 
reported.  The  principal  source  of  field  errors,  nonreporting,  declines 
sharply  over  time,  partly  because  dropouts  come  largely  from  this  group 
and  partly  because  of  pickup  from  those  remaining  in  the  panel.  For 
purposes  of  population  estimation,  the  dropouts  (panel  mortality)  may 
become  a  major  source  of  bias,  although  even  in  this  case  the  favorable 
effects  of  the  panel  serve  to  reduce  this  bias,  if  anything,  below  that  en- 
countered on  the  first  wave. 


VII.  SOURCES  OF  IMPROVEMENT 


This  chapter  brings  together  and  discusses  the  various  experiments  in- 
corporated in  project  operations  in  attempts  to  improve  the  reliability  of 
the  survey  data.  These  experiments  are  classified  under  five  general  head- 
ings, which  also  serve  as  the  first  five  sections  of  this  chapter.  The  first 
heading  refers  to  the  type  of  data  sought,  and  the  second  encompasses  the 
forms  and  instruments  used  in  data  collection.  Excluded  from  this  second 
heading,  however,  is  the  subject  of  interviewer  selection,  training,  and 
supervision  which,  because  of  its  importance,  merits  separate  treatment 
under  a  heading  of  its  own,  the  third  part  of  the  chapter.  Other  controls 
that  might  be  imposed  on  the  survey  operation  provide  the  subject  matter 
for  the  fourth  section,  perhaps  most  important  of  which  are  means  of 
motivating  respondents  to  provide  more  accurate  and  complete  informa- 
tion. The  fifth  section  is  devoted  to  different  methods  of  evaluating  the 
reliability  with  which  data  are  reported. 

The  sixth  part  of  this  chapter  is  devoted  to  a  general  review  of  means 
of  data  improvement  other  than  better  survey  methods.  Although  still 
relatively  rare,  approaches  of  this  type  would  seem  to  be  indispensable 
for  surveys  on  a  subject  such  as  consumer  saving.  Reasons  for  this  view, 
and  two  approaches  developed  in  the  course  of  the  Consumer  Savings 
Project,  are  outlined  in  this  section,  which  is  then  followed  by  a  chapter 
summary. 

Data  Sought 

Either  holdings  or  change  data  are  invariably  sought  on  financial  sur- 
veys. The  manner  in  which  these  types  of  data  were  sought  on  the  differ- 
ent surveys  of  the  Consumer  Savings  Project,  and  the  results  obtained, 
have  already  been  discussed  in  preceding  chapters.  Hence,  some  supple- 
mentary and  summary  comments  would  seem  sufficient  at  this  point. 

Holdings  Information 

Three  different  means  of  collecting  holdings  information  have  been 
investigated. 
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Table  73.   Refusal  Rates  for  Selected  Sensitive  Assets, 
Wave  1,  Studies  PI  to  P3 


Asset 


Study 


Percentage  of  respondents  refusing 


Dollar 
amounts 


Name  of 
institution 


Both 


Savings  accounts. 


Life  insurance. 


Stocks  and  nongovernment  bonds 


United  States  savings  bonds. 


Real  estate  (other  than  own 
home) 


PI 
P2 
P3 

PI 
P2 
P3 

PI 
P2 
P3 

PI 
P2 
P3 

PI 
P2 
P3 


20.8 

13.6 

4.1 


15.6 

14.3 

1.8 

20.8 
7.0 
3.5 

24.0 

19.7 

7.0 

8.3 
5.7 
0.6 


8.3 
6.4 
1.2 

19.8 
7.1 
0.0 

16.6 
4.2 
1.8 


12.5 
5.0 
0.0 

15.6 
4.2 
1.8 


Names  and  Detailed  Breakdowns 

For  validation  data  to  be  most  useful,  dollar  balances  must  be  obtained 
on  each  holding,  and  each  must  be  identified  by  institution.  Such  names 
and  individual  listings  may  also  be  sought  for  reasons  of  substantive  anal- 
ysis, such  as  to  investigate  the  concentration  by  institution  of  holdings  of 
a  particular  asset  or  the  geographic  dispersion  of  consumer  portfolios. 
For  all  of  these  reasons,  detailed  information  on  each  holding  was  sought 
on  each  panel  operation.  These  details  included  the  name  in  which  the 
asset  was  registered,  the  exact  dollar  balance,  the  name  of  the  institution 
and,  in  some  instances,  the  date  of  the  last  transaction.  Of  these  details, 
the  dollar  balance  and  the  name  of  the  institution  are  the  most  personal 
and  are  hence  likely  to  be  highly  sensitive. 

Nevertheless,  the  panel  experience  suggests  that  the  great  majority  of 
those  who  cooperated  were  willing  to  provide  this  information,  although 
the  request  for  these  data  did  produce  a  somewhat  higher  refusal  rate  than 
might  have  been  obtained  otherwise.  The  first  of  these  statements  is 
supported  by  Table  73,  which  shows  item  refusal  rates  for  the  more  sen- 
sitive assets  on  the  first  wave  of  three  of  the  panel  operations.  Moreover, 
a  certain  proportion  of  those  who  refused  these  items  of  information  on 
the  first  wave  did  give  these  data  on  later  waves.  Thus,  for  dollar  balances 
on  savings  accounts,  this  was  true  of  30  percent,  53  percent,  and  14  per- 
cent of  the  item  refusals  on  Studies  PI,  P2,  and  P3,  respectively.  Virtually 
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the  same  percentages  (and  the  same  people)  also  later  gave  names  of 
institutions.  As  a  general  rule,  asking  for  the  name  of  the  institution  caused 
no  additional  friction  once  the  dollar  balance  had  been  reported. 

That  requests  for  these  details  produced  a  somewhat  higher  refusal  rate 
than  might  otherwise  have  resulted  can  be  inferred  from  experience  with 
questionnaires  on  which  such  data  were  not  sought  and  from  an  analysis 
of  the  reasons  for  refusals.  As  has  been  noted  in  previous  chapters,  total 
refusals  can  be  segregated  by  those  who  refused  to  grant  an  interview  and 
by  those  who  granted  an  interview  but  refused  to  give  any  financial  data. 
For  the  former  group  the  request  for  details  is  clearly  not  relevant,  be- 
cause sample  members  were  not  told  exactly  what  financial  data  would 
be  sought  until  after  the  interview  began.134 

Among  the  item  refusals,  the  predominant  reason  appears  to  have 
been  the  reluctance  to  provide  detailed  information.  This  is  borne  out  in 
part  by  reports  of  the  interviewers  and  in  part  by  the  following  two  com- 
parisons: The  first  relates  to  the  change  forms  of  Wave  1  of  Studies  P2 
and  P3,  on  which  sample  members  were  first  asked  for  overall  aggregate 
dollar  balances  and  were  later  asked  for  changes  in  holdings  for  each 
asset  individually.  This  comparison  may  be  vitiated  by  the  fact  that 
change  information  tends  to  be  less  sensitive  than  dollar  balances,  but  this 
lower  sensitivity  of  the  change  data  should  be  more  than  offset  by  the 
request  for  itemization  and  for  names  of  institutions.  The  refusal  rates 
on  the  overall  aggregates  for  savings  accounts  are  3  percent  and  2  percent 
on  Studies  P2  and  P3,  respectively,  whereas  the  refusal  rates  on  dollar 
figures  (or  names  of  institutions)  item  by  item  are  14  percent  and  4  per- 
cent, respectively.  The  much  greater  emphasis  placed  on  dealing  with 
item  refusals  in  the  farm  panel  was  undoubtedly  at  least  partly  responsible 
for  the  sharp  decline  from  Study  P2  to  Study  P3  in  item  refusals  relative 
to  refusals  on  overall  aggregates. 

Second,  it  is  worth  noting  that  item  refusals  as  a  proportion  of  total 
refusals  on  .savings  account  dollar  balances  on  Wave  1  of  Studies  PI,  P2, 
and  P3  were  33  percent,  41  percent,  and  25  percent,  respectively.  If  all 
of  the  item  refusals  were  caused  by  the  request  for  this  information  rather 
than  for  dollar  aggregates,  the  total  refusals  would  have  been  reduced 
somewhat.  However,  judging  by  the  preceding  comparison,  a  substantial 
proportion  of  the  item  refusals  would  have  undoubtedly  refused  dollar 
aggregates. 

Segmentation  of  the  Portfolio 

Another  approach  to  improving  data  reliability  is  to  cover  fewer  sub- 

134  The  only  exception  was  that  interviewers  were  permitted  to  mention  what 
types  of  data  would  be  requested,  although  here,  too,  no  reference  was  to  be  made 
to  details  nor  were  the  respondents  to  be  shown  the  questionnaire. 
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Table  74.  Data  Reported  by  Number  of  Assets  Covered 

in  Interview,  by  Change  Versus  Holding  Form, 

Waves  1  and  2,  Study  P2 


Category- 


Holdings  form 


All 


Some 


Change  form 


All 


Some 


Percentage  of  respondents  reporting 
Assets  on  both  forms  on  Wave  1 

Savings  accounts 

Life  insurance 

Assets  omitted  on  Wave  1  from  "Some"  forms'4 

United  States  savings  bonds 

Marketable  bonds  and  common  and 

preferred  stock 

Number  of  holdings  reported  per  respondent 

Savings  accounts 

Life  insurance 

United  States  savings  bonds 

Marketable  bonds  and  common  and  preferred 
stock 

a  Based  on  Wave  2  reports. 


100 
82 

65 
37 

1.9 

3.5 
24.9 

4.1 


100 
94 

35 
18 

2.3 

3.3 

13.4 

2.4 


78 
6 

19 


1.6 
2.0 
3.8 

2.0 


86 

11 

21 
16 

1.8 
1.3 
4.3 

1.8 


jects  on  a  particular  interview.  This  hypothesis  was  tested  in  Waves  1 
and  2  of  Study  P2  by  questioning  half  of  the  sample  members  about  their 
total  consumer  portfolio,  with  the  questioning  focusing  on  dollar  balances 
or  change,  depending  on  the  particular  approach.  For  the  other  half  of 
the  sample  only  certain  assets  were  covered  on  the  first  wave,  whereas  the 
entire  portfolio  was  covered  on  the  second  wave.  The  assets  omitted  on 
the  first  wave  were  United  States  savings  bonds,  marketable  bonds,  com- 
mon and  preferred  stock,  and  own  business.  If  the  basic  hypothesis  is 
correct,  assets  covered  on  both  forms  should  be  reported  more  frequently 
when  some  assets  are  omitted,  although  the  differential  may  be  greater 
on  the  holdings  form  than  on  the  change  form  because  the  tendency  not 
to  report  particular  assets  if  there  was  no  change  could  camouflage  any 
relationships  in  the  latter  instance.  On  the  other  hand,  assets  omitted 
on  Wave  1  should  be  reported  with  much  the  same  frequency  on  Wave  2 
for  that  half  of  the  sample  as  they  would  be  for  the  other  half  of  the 
sample  questioned  about  the  entire  portfolio  on  Wave  l.135 

The  data  in  Table  74  provide  only  partial  support  for  this  hypothesis. 
More  frequent  reports  of  holdings  on  the  forms  covering  selected  assets  do 
result  on  all  the  change  comparisons,  although  the  number  of  holdings 

135  The  reason  for  this  it  that  respondents  who  had  not  been  questioned  about 
these  assets  on  Wave  1  were  questioned  about  not  only  these  assets  but  the  entire 
portfolio  on  Wave  2.  By  hindsight,  it  might  have  been  more  effective  had  the 
assets  covered  for  this  group  been  such  that  part  of  the  portfolio  was  covered  on 
Wave  1  and  the  other  part  on  Wave  2,  with  no  overlap  in  coverage. 
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per  respondent  differs  little  on  either  form.  On  the  holdings  form,  how- 
ever, only  life  insurance  is  reported  more  frequently  with  more  limited 
asset  coverage,  whereas  reports  of  the  dollar  values  of  two  assets  are  far 
higher  when  asked  for  in  conjunction  with  all  other  assets  on  the  first  wave 
than  they  are  when  questions  about  these  holdings  are  postponed  to  the 
second  wave.  The  number  of  holdings  reported  tends  also  to  be  much 
larger  in  the  former  case.  Thus,  these  results  would  suggest  that  restriction 
of  the  scope  of  the  questionnaire  produces  better  reporting  of  the  assets 
included  initially,  but  at  the  expense  (at  least  on  the  holdings  form)  of 
assets  not  covered  until  the  second  wave.  Possibly  this  limitation  might  not 
exist  if  assets  already  covered  in  the  first  wave  were  excluded  from  the 
second  wave  questionnaire. 

Overall  Total  Versus  Detailed  Breakdowns 

Whether  or  not  more  complete  data  are  obtained  by  asking  for  de- 
tailed breakdowns  than  by  requesting  overall  totals  was  tested  on  Study 
P2  and  on  Study  P33  each  in  a  slightly  different  form.  On  the  first  wave 
of  Study  P2,  all  sample  members  were  asked  initially  for  overall  assets  and 
for  total  holdings  in  dollar  brackets  for  each  of  10  asset  categories  and 
for  total  asset  holdings.136  Then,  the  various  experimental  approaches, 
explained  in  previous  chapters,  were  applied.  Specifically,  half  of  the 
sample  members  were  asked  for  detailed  holdings  information  and  the 
other  half  were  asked  for  detailed  change  information.  As  a  result,  com- 
parisons can  be  made  between  the  frequency  of  reporting  and  the  asset 
size  distributions  for  the  two  types  of  data  on  the  holdings  form,  as  well 
as  between  the  statistics  for  the  holdings  form  and  the  comparable  aggre- 
gate estimates  obtained  from  the  change  form. 

If  the  financial  data  are  more  complete  when  detailed  breakdowns  are 
requested,  as  is  often  supposed,  the  frequency  of  reporting  of  particular 
assets  from  the  detailed  breakdowns  should  be  higher  than  from  the  aggre- 
gate estimates.  Also,  one  would  expect  the  size  distribution  of  the  total 
holdings  of  each  asset  as  obtained  from  the  detailed  breakdowns  to  be 
more  skewed  to  the  right  than  the  size  distribution  as  obtained  from  the 
aggregate  estimates.  In  other  words,  omissions  would  be  less  likely  on  the 
detailed  questions,  so  that  total  holdings  would  be  larger  in  the  former 
case  than  in  the  latter.  Similarly,  one  would  expect  the  size  distribution 
of  total  holdings  of  a  particular  asset  as  obtained  from  the  detailed  break- 
downs to  be  more  right-skewed  than  the  size  distribution  of  total  holdings 
obtained  from  the  overall  estimates  on  the  change  form,  which  in  turn 

130  The  asset  categories  were  checking  accounts,  savings  accounts,  life  insur- 
ance, United  States  savings  bonds,  marketable  bonds,  own  business,  common  and 
preferred  stock,  trust  funds  and  mortgages,  loans  lent,  and  own  home  and  other 
real  estate. 
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would  not  be  expected  to  be  significantly  different  from  the  comparable 
distribution  obtained  from  the  holdings  form. 

A  similar  test  in  Study  P3  consisted  of  asking  respondents  on  the 
change  form  initially  for  overall  estimates  of  holdings,  but  not  asking  them 
for  such  overall  estimates  on  the  holdings  form.  As  a  result,  the  test  is 
weaker  than  before,  although  the  criteria  are  much  the  same.  In  other 
words,  a  lower  rate  of  nonreporting  and  a  more  right-skewed  distribution 
of  holdings  from  the  detailed  information  on  the  holdings  form  than 
from  the  overall  estimates  on  the  change  form  provide  support  for  this 
hypothesis. 

Comparison  of  ownership  rates  for  the  different  assets  on  the  two 
studies,  by  form  and  by  approach,  brings  out  virtually  no  differences  in 
the  frequency  of  reporting  assets  as  a  result  of  using  the  different  ap- 
proaches. Differences  are  slight  and  fail  to  exhibit  any  consistent  pattern. 
Comparison  of  size  distributions  for  each  study  for  five  types  of  holdings 
—  checking  accounts,  savings  accounts,  life  insurance,  common  and  pre- 
ferred stock,  and  total  assets  —  did  bring  out  some  interesting  points. 
Virtually  no  differences  in  the  distributions  of  checking  accounts  and 
savings  accounts  are  observed  from  the  results  of  using  the  various  ap- 
proaches. However,  larger  holdings  were  reported  more  frequently  on  the 
detailed  breakdowns  in  the  case  of  life  insurance  and  stock  (Table  75). 
This  tendency  is  most  pronounced  in  the  case  of  total  assets,  which  in- 
cludes the  four  categories  mentioned  previously,  and  savings  bonds,  mar- 
ketable bonds,  and  own  business  —  particularly  on  Study  P2  in  which 
nearly  two-thirds  reported  total  assets  of  $25,000  or  more  on  the  detailed 
breakdowns  as  compared  with  one-half  of  those  asked  for  overall  estimates 
on  the  change  form.  Thus,  by  these  criteria,  seeking  detailed  breakdowns 
of  financial  holdings  would  seem  to  yield  more  complete  information. 

Saving  Estimates 

Saving  could  be  estimated  by  asking  directly  for  change  in  the  stock 
of  savings  or  by  asking  for  the  amount  of  savings  at  two  different  points 
in  time.  Chapter  VI  showed  that  in  several  instances  in  which  comparison 
was  possible,  saving  in  the  form  of  computed  change  was  at  least  as  ac- 
curate as  estimates  obtained  directly,  and  it  was  often  much  more  accurate. 
Indeed,  in  the  latter  case  a  question  may  be  raised  as  to  whether  or  not 
some  estimates  of  saving  should  be  sought  at  all  in  this  manner  because 
of  the  substantial  errors  that  were  observed. 

Yet,  the  results  of  these  and  other  studies  suggest  that  estimates  of 
saving  in  the  form  of  computed  change  may  also  contain  substantial 
errors.  In  particular,  dollar  balances  obtained  as  of  two  different  points 
in  time  during  the  same  interview  appear  to  yield  highly  inaccurate  esti- 
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mates  of  saving.  Such  a  result  was  obtained  in  the  Netherlands  reliability 
survey  (Chapter  IV)  and  also  in  Study  S3.  In  other  words,  if  only  one 
interview  is  possible  with  a  family,  estimates  of  saving  may  be  subject  to 
substantial  response  error  regardless  of  which  method  is  used. 

Saving  and  Savings  Data  Requested  Simultaneously 

On  Wave  3  of  Study  PI  and  Wave  4  of  Study  P3,  respondents  were 
requested  to  provide  both  dollar  balances  and  changes  in  holdings  occur- 
ring since  the  time  of  the  last  interview.  Although  this  procedure  was  not 
tested  on  an  experimental  basis,  the  results  provide  no  basis  for  question- 
ing its  feasibility.137  No  instances  were  reported  of  respondents  refusing 
to  provide  both  types  of  information,  although  some  respondents  noted 
that  the  change  could  be  computed  as  the  difference  between  the  dollar 
balances  reported  on  the  current  and  preceding  interviews.  On  the  other 
hand,  it  should  be  noted  from  the  preceding  chapter  that  computed  saving 
still  was  considerably  more  accurate  than  reported  saving. 

Data-Collection  Methods 
Structured  Versus  Unstructured  Approach 

Two  experiments  concerning  the  structure  of  the  approach  were 
carried  out.  As  was  noted  in  Chapter  III,  a  highly  structured  question- 
naire was  used  on  a  randomized  part  of  the  sample  members  in  Study  PI, 
and  a  completely  unstructured  approach  was  used  on  the  remainder. 
However,  this  experiment  was  not  fully  controlled  because  different  inter- 
viewers were  used  with  each  type  of  approach.  In  Study  P2,  half  of  the 
sample  members  were  interviewed  by  means  of  a  structured  questionnaire 
form,  and  the  other  half  were  interviewed  with  the  aid  of  reminder  cards 
pasted  into  the  interviewer's  notebook  but  without  any  formal  question- 
naire. This  experiment  was  under  statistical  control,  with  randomization 
of  both  interviewers  and  sample  members. 

The  results  of  these  two  experiments  are  summarized  in  Table  76, 
based  on  the  use  of  three  criteria  —  the  rate  of  refusal,  the  frequency  with 
which  ownership  of  particular  assets  was  reported,  and  the  frequency  with 
which  complete  details  were  obtained  from  reported  owners.  As  is  evident 
from  this  table,  similar  differences  resulted  on  both  studies.    In  Study  PI, 

137  The  question  layout  used  for  this  purpose  is  reproduced  below: 

_  Change  since 

Present         

balance  Increase  Same  Decrease 

This  particular  form  was  found  to  be  much  more  effective  in  asking  for 
change  than  the  usual  single-column  arrangement  because  it  differentiates  clearly 
between  increases  and  decreases. 
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Table  76.  Response  Rates  and  Asset  Ownership  Reported 
by  Type  of  Questionnaire,  Studies  PI  and  P2 


Study  PI 

Study  P2 

Statistic 

Struc- 
tured 

Unstruc- 
tured 

Ques- 
tionnaire 

Reminder 
card  only 

Number  of  contacts 

180 

72 

80 
93 
57 
60 
25 

82 
83 
91 
82 
84 

76 
87 

70 
88 

42 

64 

9 

59 
49 
10 
48 
25 

197 

77 

90 
88 

60 
31 
29 

65 

76 
82 
51 
81 
96 

171 

Response  rate  (percent) 

Percentage  of  respondents  reporting 
ownership  of 
Savings  accounts 

82 
100 

Life  insurance 

88 

United  States  savings  bonds 

Stock  and  other  marketable  bonds 
Real  estate  other  than  own  home .  . 
Percentage  of  validated  savings 

accounts  reported 

Percentage  of  assets  reported  with 
full  details 

Savings  accounts 

Life  insurance 

55 
30 
24 

58 

72 
52 

United  States  savings  bonds 

Stock  and  other  marketable  bonds 
Real  estate  other  than  own  home .  . 

40 
87 
96 

a  much  higher  response  rate  was  obtained  with  the  unstructured  approach. 
Yet  by  each  of  the  other  two  criteria  the  structured  questionnaire  appears 
to  be  far  superior.  Not  only  is  each  of  the  assets  listed  reported  with  much 
higher  frequency  on  the  structured  questionnaire  than  it  is  when  the  un- 
structured approach  is  used,  but  in  each  case  a  much  larger  proportion  of 
the  assets  that  were  reported  contain  full  details.138  Thus,  the  completeness 
of  reporting  with  the  structured  approach  ranged  between  80  and  90  per- 
cent, whereas  with  the  unstructured  approach  the  highest  completeness 
rate  was  59  percent  and  it  fell  as  low  as  10  percent  for  United  States  sav- 
ings bonds.  In  Study  P2  response  was  also  higher  when  no  formal  ques- 
tionnaire was  used,  but  in  the  two  other  respects  the  structured  approach 
seems  to  have  been  superior. 

The  fact  that  these  two  approaches  were  not  randomized  among  inter- 
viewers in  Study  PI  (although  they  were  randomized  among  sample  mem- 
bers) is,  if  anything,  another  point  in  favor  of  the  structured  approach. 
The  reason  is  that  only  interviewers  who  had  had  some  experience  with 
unstructured  questionnaires  and  who  were  willing  to  undertake  the  neces- 
sary memory  work  were  assigned  to  the  unstructured  approach.    These 

138  By  "full  details"  is  meant  definite  responses  to  a  list  of  items  requested  for 
each  asset.  Thus,  for  savings  accounts,  this  list  included  name  of  the  account, 
name  of  the  institution,  date  of  last  transaction,  and  dollar  balance  as  of  the  date 
of  the  interview. 
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interviewers  were  originally  rated  at  least  as  able  as  the  interviewers  using 
the  structured  questionnaire. 

At  the  same  time,  the  pronounced  differences  between  the  two  ap- 
proaches suggest  another  explanation  largely  independent  of  interviewer 
quality.  This  explanation,  advanced  in  previous  chapters,  is  that  the  more 
rigorous  structured  questionnaire  approach  served  to  weed  out  people 
who  were  inclined  to  be  uncooperative  anyway,  with  the  result  that  sample 
members  who  did  grant  interviews  were  basically  more  cooperative  than 
those  interviewed  with  the  unstructured  questionnaire.  This  explanation 
receives  support  from  the  experiment  in  Study  P2,  also  shown  in  Table  76. 

The  "Warm-up"  Hypothesis 

It  has  sometimes  been  conjectured  that  more  complete  financial  infor- 
mation would  be  obtained  on  consumer  surveys  if  the  request  for  such 
data  were  postponed  to  a  later  stage  of  the  interview  or,  in  the  case  of  a 
panel  operation,  to  a  later  interview.  Both  forms  of  this  hypothesis  were 
tested  in  Study  PI  by  the  factorial  method.  As  was  explained  in  earlier 
chapters,  three  different  split-run  experiments  were  superimposed  on  each 
other  on  the  first  wave  of  this  study  —  a  structured  questionnaire  versus 
an  unstructured  questionnaire,  requesting  dollar  balances  and  names  of 
institutions  for  all  holdings  on  the  first  wave  versus  seeking  this  informa- 
tion only  on  the  second  wave,  and  asking  on  the  first  wave  for  all  details 
versus  not  asking  for  dollar  balances  or  names  of  institutions. 

The  third  experiment  was  carried  out  by  splitting  the  two  structured 
questionnaire  subsamples  into  halves  once  more  and  asking  half  of  this 
group  first  for  data  on  asset  holdings  and  then  for  data  on  debt,  and 
following  the  reverse  procedure  with  the  other  half.  Presumably,  respon- 
dents would  be  more  reluctant  to  give  information  on  assets  than  on  debts. 
Hence,  if  the  "warm-up"  hypothesis  held  in  Study  PI,  more  assets  should 
have  been  reported  by  the  respondents  when  this  information  was  sought 
last  than  when  it  was  sought  first. 

On  the  whole,  the  hypothesis  is  supported  by  the  data  (Table  77) .  A 
higher  proportion  of  respondents  reported  each  of  the  five  assets  listed 
when  debts  were  covered  first  than  the  proportion  when  assets  were 
covered  first.  Some  of  the  differences  are  substantial,  and  statistically  sig- 
nificant, such  as  for  life  insurance  and  for  stock  and  marketable  bonds. 
In  addition,  the  number  of  holdings  reported  per  respondent  when  debts 
were  covered  first  is  much  greater  than  the  number  when  assets  were 
covered  first;  this  was  particularly  noticeable  for  savings  bonds  and  for 
stock  and  marketable  bonds. 

Results  pertaining  to  the  panel  application  of  the  "warm-up"  hy- 
pothesis do  not  provide  much  support  for  the  desirability  of  a  "warm-up" 
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Table  77.  Data  Reported,  by  Order  of  Coverage  of  Assets  and 
Debts  on  Structured  Questionnaires,  Wave  1,  Study  PI 


Statistic 


Assets  first 

Debts  first 

78 

82 

88 

98 

53 

62 

50 

71 

24 

26 

1.4 

1.7 

4.0 

3.9 

5.4 

10.0 

1.9 

2.9 

.4 

.3 

Percentage  of  respondents  reporting 

Savings  accounts 

Life  insurance 

United  States  savings  bonds 

Stock  or  marketable  bonds 

Real  estate  other  than  own  home 

Number  of  holdings  reported  per  respondent 

Savings  accounts 

Life  insurance 

United  States  savings  bonds 

Stock  or  marketable  bonds 

Real  estate  other  than  own  home 


period  before  requesting  sensitive  information.  The  main  evidence  in  sup- 
port of  this  hypothesis  is  the  higher  frequency  with  which  savings  accounts 
and  life  insurance  were  reported  on  the  structured  questionnaire  approach 
when  dollar  balances  were  not  requested,  and  the  lower  refusal  rate  on 
the  unstructured  questionnaire  approach  when  no  dollar  information  was 
sought.  However,  only  the  first  of  these  findings  tends  to  support  the  hy- 
pothesis, because  most  refusals  would  have  been  encountered  before  dollar 
figures  were  requested.  On  the  other  hand,  other  assets  were  reported  as 
frequently  when  one  approach  was  used  as  when  the  other  was  used. 
Indeed,  reporting  of  stock  seems  to  have  been,  if  anything,  considerably 
higher  when  dollar  figures  were  requested,  and  much  the  same  was  true 
of  United  States  savings  bonds. 

A  further  comparison  of  the  two  approaches  was  made  on  the  basis 
of  the  frequency  with  which  holdings  were  reported  and  with  which  dollar 
amounts  were  given  by  the  various  subsamples  after  the  first  two  waves 
of  interviews  on  Study  PI.  By  that  time,  figures  and  names  of  institutions 
had  been  requested  of  all  of  the  sample  members.  Hence,  if  the  "warm- 
up"  period  were  really  effective,  the  frequency  with  which  holdings  and 
dollar  balance  were  reported  should  have  been  higher  among  those  not 
asked  initially  for  dollar  figures. 

The  refusal  rate  was  indeed  lowest  on  the  unstructured  approach  when 
no  dollar  amounts  were  requested,  and  the  highest  frequencies  of  holdings 
were  obtained  with  this  approach.  However,  complete  details  on  holdings 
were  more  likely  to  be  obtained  with  a  structured  questionnaire  seeking 
dollar  balances. 

Varying  the  Interview  Interval 

The  present  studies  provide  two  series  of  tests  relating  to  length  of 
interval  between  interviews.   One  series  of  tests,  presented  in  Chapter  VI, 
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related  to  the  accuracy  with  which  the  panel  members  of  Study  P3  re- 
ported debt,  checking  accounts,  and  savings  accounts  when  half  of  the 
members  were  interviewed  on  every  wave  and  the  other  half  were  inter- 
viewed on  every  other  wave.  The  findings,  it  will  be  recalled,  were  in- 
conclusive, suggesting  that  more  accurate  reporting  of  debt  was  produced 
by  the  more  frequent  interviews  than  by  the  less  frequent  interviews. 
However,  the  same  phenomenon  was  not  in  evidence  for  checking  or  sav- 
ings accounts. 

It  may  be  argued  that  the  discriminatory  power  of  this  first  series  of 
tests  is  not  very  great  because  the  comparisons  involved  intervals  fairly 
close  to  one  another.  To  some  extent  this  deficiency  can  be  remedied  by 
the  second  series  of  tests,  summarized  in  Table  78.  The  comparisons  which 
form  the  basis  for  this  table  relate  to  the  estimation  of  saving  in  a  partic- 
ular form  by  summing  the  reports  of  savings  accounts  obtained  from  con- 
secutive interviews  with  an  estimate  of  saving  obtained  as  the  difference 
between  holdings  reported  at  the  start  of  the  period  and  holdings  reported 
at  the  end  of  the  period.  Two  such  sets  of  comparisons  are  presented, 
one  for  savings  accounts  for  Studies  P2  and  P3,  and  one  for  three  different 
types  of  farm  assets  for  Study  P3.  In  the  case  of  savings  accounts,  one  set 
of  estimates  of  saving  was  obtained  as  the  difference  between  holdings 
reported  on  Wave  1  and  holdings  reported  on  Wave  5,  using  only  those 
sample  members  in  each  study  who  reported  holdings  on  both  waves.  The 
other  set  of  estimates  was  derived  as  the  cumulated  sum  of  saving  in  the 
form  of  savings  accounts,  based  on  those  sample  members  interviewed  on 
every  wave  and  reporting  this  information.  For  each  comparison,  corre- 
sponding saving  estimates  computed  from  the  institutional  data  serve  as 
the  yardstick  of  accuracy. 

The  saving  comparisons  relating  to  farm  assets  are  of  a  somewhat  dif- 
ferent nature.  In  this  case,  one  comparison  is  based  on  replies  obtained 
from  respondents  on  Wave  5  of  Study  P3,  conducted  in  January,  1962, 
in  which  estimates  were  requested  of  total  saving  in  the  form  of  land  and 
buildings,  livestock,  and  machinery  for  the  entire  year  1961.  A  second 
comparison  was  obtained  by  cumulating  reports  of  saving  of  these  assets 
provided  by  those  members  of  the  farm  panel  interviewed  on  every  wave. 
Here,  the  comparison  relates  to  the  same  panel  members  in  both  instances 
because  no  objective  measure  of  accuracy  is  available. 

To  judge  by  Table  78,  the  results  are  still  not  conclusive.  The  more 
accurate  estimate  of  saving  in  the  form  of  savings  accounts  varies  with  the 
study,  although  the  small  sample  sizes  and  the  unstable  nature  of  the 
averages  in  Study  P3  lend  greater  credence  to  the  Study  P2  comparison. 
In  that  case,  more  accurate  estimates  are  obtained  by  differencing  the  bal- 
ances reported  at  the  start  and  end  of  the  panel  operation.    The  results 


226     The  Reliability  of  Consumer  Reports  of  Financial  Assets  and  Debts 


Q 

O 

<£ 

W 

PL, 

c* 

< 

w 

> 

w 

2 

0 

<! 

« 

> 

0 

CO 

Cm 

Q 

CO 

2 

H 

< 

r/) 

CM 

w 

&H 

< 

in 

S 

W 

<& 

W 

< 

i-) 

Uh 

H 

(/> 

2 

•— 1 

Lrt 

c 

k4 

< 
> 

cfl 

W 

H 

H 

Z 

25 

P 

1— 1 

0 

H 

u 

7, 

u 

:•■! 

< 

Pi 

w 

CO 

h 

o 

(Jh 

2 

P 

> 

■41 

05 

1/5 

u 

Ph 

£ 

<1 

1-1 

h 

n 

<! 

z 

« 

> 

0 

< 

Y, 

r/5 

Cfl 

o 

p 

r/5 

W 

H 

< 

s 

w 


"c3  s  "5  io 

Institution 
balances  o 
Wave  1  an 
on  Wave 

•  ^h       -CO       •      •      • 

•  lO       -CD       •       ■       • 

*2 

3 
be 

•  CM       ■  lO       •       ■       ■ 

">    CI 

ea  o 

S    4J 

73  s  >, 

8  v 

bc-^ 

5  8  * 

CO  CM  CD  ^h  CM  1^  CM 

■*  CD  to  <M  cs  r~  r~ 

3   u 

3   O   O 

CO*H-    1    CO  o 

So    -^ 

3  G 

3    U    4) 

HfH        t-i  CM 

3    ft    > 

&^        1               1 

s-,   be 
rt   3 

0£t 

1                1 

'"0 

2   3 

v^ 

bo 

oj    nj    O     . 

S-C 

fferenc 
een  W 
d  Wav 
alances 

co  co  ■r-i  c£>  io  r>.  eo 

V 
> 

O  CM  m  iO  CT)  CT)  m 

cm  co  r-»  m  CD  ■*  co 

5BC^ 

m 

V 

J2^H 

1) 

P-,  13 

6-1 

CM  •*  Tj"  LO  tx  H  CO 

tD  ■*  t-h         <*  CD  CO 

rt  M 

03 

-M              +-> 

3         3 

3         3 

O         O 

1) 

u       o 

be 

c«  u  m  y 

<a 

™  ««  « 

S-. 

u 

> 
o 
U 

All  accoun 
Validated 
All  accoun 
Validated 

>- 

-a 

CM         CO         CO  CO  CO 

3 

Cm        Cm        Cm  Oh  Cm 

W 

be 

C 

> 

rt 

Cm 

O 

CA 

§ 

bo 

3 

o 

ings  account 
id  and  build 

V 

.a 

-3 

u 

>                   >-   >   «S 

1                    n 

1                        H 

1  .- 

2 

Sources  of  Improvement  227 

yielded  by  this  method  for  saving  in  farm  assets  seem  also  more  plausible 
than  the  negative  saving  obtained  by  the  other  method,  since  the  period 
covered  by  that  study  was  one  of  growing  prosperity. 

Mail  Questionnaires  Versus  Personal  Interviews 

Substantial  economy  could  be  achieved  if  financial  data  were  secured 
by  mail.  The  practicability  of  this  procedure,  however,  is  very  much  an 
open  question  in  two  respects.  First,  the  cost  of  a  mail  questionnaire  re- 
mains to  be  established,  that  is,  whether  or  not  enough  sample  members 
will  return  the  questionnaire  to  warrant  use  of  the  mail  technique.  Second, 
are  data  supplied  by  mail  as  reliable  as  data  obtained  in  a  personal  inter- 
view? If  the  reliability  of  the  mail  questionnaire  data  is  much  lower,  the 
technique  may  be  of  little  value,  regardless  of  the  rate  of  return. 

The  use  of  a  mail  questionnaire  was  tested  on  three  of  the  panel  opera- 
tions of  the  project.  In  each  case  the  test  was  carried  out  on  the  fourth 
wave  of  interviews,  so  that  the  results  cannot  be  used  to  judge  the  feasi- 
bility of  mail  questionnaires  in  a  one-time  operation.  However,  if  the 
mail  questionnaire  is  not  feasible  on  a  later  wave  of  a  panel  study  it  would 
certainly  not  be  feasible  in  a  one-time  operation. 

In  Study  PI,  a  mail  questionnaire  was  sent  at  the  beginning  of  the 
fourth  wave  to  all  still-active  panel  members.  Two  types  of  mail  question- 
naires were  used,  being  allocated  to  the  panel  members  on  a  split-run  basis. 
One  questionnaire  was  relatively  short,  contained  only  two  pages,  and 
requested  changes  in  holdings  since  the  last  interview.  The  second  ques- 
tionnaire was  longer,  covered  four  pages,  and  sought  dollar  balances  on 
such  holdings  as  checking  accounts,  savings  accounts,  and  debt,  and 
changes  on  such  assets  as  common  stock,  life  insurance,  and  real  estate. 

All  together,  47  percent  of  the  178  regular  mail  questionnaires  were 
returned.139  Those  not  returning  mail  questionnaires  were  contacted  by 
personal  interview,  and  interviews  were  obtained  with  82  percent.  The 
response  on  the  short  form  was  somewhat  higher,  although  not  significantly 
so,  than  on  the  longer  form  —  51  percent  versus  43  percent. 

In  Studies  P2  and  P3,  mail  questionnaires  were  sent  to  a  randomized 
half  of  the  cooperative  and  uncooperative  panel  members  on  Wave  4.  In 
Study  P2,  the  mail  questionnaire  contained  four  pages  with  virtually  the 
same  content  as  the  long  mail  questionnaire  in  Study  PI.  The  mail  ques- 
tionnaire in  Study  P3  contained  three  pages  and  focused  on  changes  in 
holdings  only. 

139  In  addition,  short  mail  questionnaires  were  sent  to  17  "hard"  refusals  on 
the  third  wave  and  to  five  panel  members  who  had  indicated  on  earlier  waves  a 
preference  for  supplying  data  by  mail.  The  returns  from  these  two  groups  were 
one  and  two,  respectively. 
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Table  79.  Completeness  of  Reporting,  by  Mail  Return  Versus 
Personal  Interview,,  Studies  P2  and  P3 


Study 

Holding 

Percentage  of  known 

holders  supplying 

all  details 

Percentage  of  dollar 

figures  given  to 

nearest  cent 

Mail 

Personal 
interview 

Mail 

Personal 
interview 

P2 

P3 

Checking  accounts 
Savings  accounts 
Personal  debt 

Checking  accounts 
Savings  accounts 
Personal  debt 
Farm  debt 

92 
93 
82 

94 
94 
92 
88 

97 
93 
94 

100 
100 
100 
100 

53 
58 
44 

47 
27 
55 
40 

44 
48 
27 

36 
15 
13 
18 

The  rates  of  return  of  the  mail  questionnaires  on  Study  P2  were  41  per- 
cent among  the  cooperative  panel  members  and  26  percent  among  the 
uncooperative  ones.  On  Study  P3,  62  percent  of  both  the  cooperative 
and  the  uncooperative  panel  members  returned  the  mail  questionnaire. 
On  both  studies,  cooperative  panel  members  not  returning  mail  question- 
naires were  assigned  to  personal  interviewers,  as  was  done  in  Study  PI. 
Such  interviews  were  successfully  carried  out  with  95  percent  of  such 
people  on  Study  P2  and  with  all  of  those  on  Study  P3.  At  the  same  time, 
it  should  be  noted  that  7  percent  of  the  panel  members  sent  mail  question- 
naires on  Study  P2  refused  to  cooperate  any  further,  as  did  6  percent  of 
those  on  Study  P3.  Thus,  mail  returns  did  appear  to  be  substantial,  at 
least  on  a  panel  operation,  and  those  who  did  not  respond  by  mail  were 
picked  up  successfully  by  personal  interview. 

To  turn  to  the  second  principal  question  regarding  the  mail  approach, 
full  details  on  financial  holdings  were  obtained  somewhat  more  frequently 
on  the  personal  interviews,  as  is  apparent  from  Table  79.  However,  in 
terms  of  an  indicator  of  accuracy  of  the  data  —  reporting  of  dollar  figures 
to  the  nearest  cent  —  the  mail  returns  were  generally  far  superior.  In 
other  words,  the  personal  interviews  may  be  completed  more  frequently, 
but  the  accuracy  of  report  may  be  higher  on  the  mail  returns. 

Sequence  of  Questionnaire  Forms 

The  apparent  superiority  of  asking  for  holdings  rather  than  for  change 
in  a  one-time  survey  was  pointed  out  in  earlier  sections.  In  the  case  of  a 
panel  operation,  the  question  may  still  be  raised  as  to  whether  or  not  the 
two  alternative  approaches  led  to  results  different  from  those  for  a  one- 
time study.  Some  findings  relating  to  this  question  can  be  deduced  from 
Studies  P2  and  P3,  in  which  half  of  the  panel  members  were  asked  for 
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Table  80.  Effect  of  Interview  Sequence  on  Reporting 
of  Validated  Holding,  Studies  P2  and  P3 
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Holding 

Study 

Interview 
sequence 

Percentage  of  respondents 
reporting  validated  asset 

Wave  1 

Wave  3 

Wave  5 

Savings  accounts 

Debt 

P2 
P3 

P3 

CCH 
HHC 

CCH 
HHC 

CCH 
HHC 

60 
67 

58 

74 

62 
84 

79 
79 

73 
75 

88 
84 

88 
80 

80 
88 

94 

91 

change  on  the  first  two  interviews  and  for  holdings  on  the  third  interview, 
whereas  the  reverse  procedure  was  followed  with  the  other  half. 

These  findings  indicate  that  over  time  it  makes  little  difference  which 
sequence  is  used.  As  was  noted  in  Chapter  VI,  and  is  brought  out  more 
clearly  in  the  following  tabulation,  in  terms  of  response  the  use  of  the 
change  form  last  may  be  more  effective : 

Cumulative  refusal  rate 

CCH  HHC 

P2     After  one  wave 16%  22% 

After  three  waves 32  34 

After  five  waves 47  41 

P3     After  one  wave 4  10 

After  three  waves 10  22 

After  five  waves 13  24 

On  the  other  hand,  contrary  to  the  findings  pertaining  to  the  first 
wave,  reporting  of  holdings  on  later  waves  does  not  appear  to  be  any 
better  when  respondents  are  first  asked  for  change  than  for  holdings.  This 
point  is  brought  out  in  Table  80,  which  shows  that  by  the  third  interview 
the  rate  of  reporting  of  the  validated  asset  or  debt  is  more  or  less  the 
same  regardless  which  approach  has  been  used.  At  the  same  time,  since 
reporting  rates  when  respondents  are  first  asked  for  holdings  are  not  much 
lower  at  any  time  than  reporting  rates  when  respondents  are  first  asked 
for  change,  the  former  approach  would  still  seem  preferable  because  of 
the  more  uniformly  reliable  data  obtained. 


Feedback 

On  all  waves  of  the  panel  study  after  the  first,  data  previously  reported 
by  the  respondent  were  recorded  on  the  questionnaire  form  for  the  next 
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wave.  This  procedure  has  the  advantage  of  facilitating  the  conduct  of  the 
interview  and  requiring  the  respondent  to  recall  less  information. 

Consideration  was  given  to  not  supplying  this  information  on  partic- 
ular waves  or  to  withholding  particular  items  of  information,  such  as  the 
identity  of  the  family  member  having  a  particular  holding.  Such  a  with- 
holding procedure  might  serve  as  a  check  of  reliability  of  data  provided 
earlier.  Indeed,  an  even  more  complete  check  would  be  provided  if  the 
respondent  were  requested  to  report  all  of  his  holdings  over  again.  How- 
ever, this  procedure  was  not  followed  because  of  strong  resistance  from 
the  interviewers  and  because  pilot  tests  indicated  the  possibility  of  con- 
siderable respondent  irritation.  Such  a  procedure  would  yield  at  best  an 
indication  not  of  the  reliability  of  the  data  but  only  of  their  consistency, 
and  this  possibly  at  the  cost  of  a  substantial  increase  in  panel  mortality. 

Use  of  Keysort  Cards  as  Questionnaire  Forms 

In  Study  PI  all  questionnaire  forms  were  multilithed  on  white  sheets 
of  paper.  In  Studies  P2  and  P3,  the  questionnaires  were  printed  on  large 
Keysort  cards  (9'/2  x  12!/2),  partly  to  permit  certain  key  items  of  informa- 
tion to  be  coded  on  the  edges  of  the  questionnaire.  In  addition,  this  was 
done  to  indicate  to  the  respondents  how  the  data  would  be  coded  and  to 
show  them  that  individual  identification  was  not  used  in  preparing  tab- 
ulations. Interviewers  were  acquainted  with  the  coding  procedures  in  the 
training  sessions  and  were  encouraged  to  discuss  the  coding  with  the 
respondents. 

Although  this  approach  was  not  incorporated  into  a  controlled  experi- 
ment, there  seems  little  doubt  regarding  its  effectiveness.  The  interviewers 
reported  numerous  instances  of  favorable  reception  of  this  questionnaire 
card  and  of  respondents  being  left  with  a  better  impression  of  the  scientific 
nature  of  the  study.  In  some  instances,  the  questionnaire  card  was  cited 
as  a  prime  factor  in  convincing  the  sample  member  to  cooperate.  No 
unfavorable  reactions  were  reported. 

Interviewer  Control 

Since  interviewer  procedures  and  their  effects  on  the  reliability  of 
financial  data  are  the  subject  of  a  separate  monograph,  reference  can  be 
made  to  that  study140  and  relatively  little  data  need  be  presented  in  this 
section.  This  is  especially  so  because  few  controlled  experiments  could 
be  carried  out  relating  to  interviewers,  these  experiments  being  largely 
observational.    The  results  of  these  experiments  are  discussed  under  the 

140  Stanley  Steinkamp  and  Mathew  Hauck,  Survey  Reliability  and  Interviewer 
Competence,  Studies  in  Consumer  Savings,  No.  4  (Urbana,  Illinois:  University  of 
Illinois  Bureau  of  Economic  and  Business  Research,  1965). 
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usual  three-way  breakdown  of  interviewer  control  —  selection,  training, 
and  supervision. 

Selection 

Sources  of  Interviewers 

An  interviewing  staff  had  to  be  recruited  anew  for  each  panel.  In  the 
pilot  operation  in  Chicago  and  in  Study  PI,  interviewers  were  recruited 
through  usual  channels  —  newspaper  ads,  contacts  through  universities, 
and  contacts  with  interview  organizations.  State  employment  offices  were 
also  used  to  recruit  interviewers  and  were  of  considerable  assistance  both 
in  attracting  applicants  and  in  facilitating  the  interviewing  and  weeding 
out  of  applicants. 

The  best  source  of  all,  however,  was  the  local  school  districts,  a  source 
used  heavily  in  Study  P2  and  almost  exclusively  in  Study  P3.  Invariably, 
the  local  superintendent  of  schools  was  most  helpful  in  mailing  notices  of 
the  availability  of  these  interviewing  positions  to  his  teaching  staff.  They, 
as  well  as  the  teacher  associations,  seemed  particularly  willing  to  cooperate 
in  finding  applicants,  a  task  not  difficult  because  of  the  large  number  of 
teachers  always  looking  for  part-time  work. 

At  the  same  time,  teacher-interviewers  tended  to  make  very  acceptable 
and  willing  students  for  interviewer  training.  With  their  backgrounds, 
most  of  them  had  little  difficulty  mastering  interviewing  instructions  as 
well  as  the  meaning  of  financial  terms.  Moreover,  respondents  appeared 
to  be  more  willing  to  provide  confidential  data  to  interviewers  who  were 
teaching  than  to  interviewers  who  were  in  the  business  world.  This  infer- 
ence receives  support  from  the  following  tabulation,  showing  refusal  rates 
on  savings  accounts  in  Wave  1  of  Study  P2  by  the  regular  occupation  of 
the  interviewer. 

Regular  occupation  Interview  Savings  account  balance 

of  interviewer  refusal  rate  item  refusal  rate 

Teacher 18%  1.6% 

Business  or  industry 20  3.8 

Other 27  9.1 

It  should  be  stressed,  however,  that  sample  members  were  not  ran- 
domized among  interviewers.  The  lower  refusal  rates  of  the  teacher- 
interviewers  might  have  been  brought  about  by  their  higher  education  — 
and  possibly  their  greater  flexibility  or  greater  intelligence  —  than  by  their 
regular  occupation. 

Use  of  Personality  Test  in  Interviewer  Selection 

Applicants  for  interviewer  positions  on  all  three  panel  studies  were 
asked  to  take  a  personality  test  in  addition  to  filling  out  an  application 
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form,  going  through  a  personal  interview,  and  providing  references.  On 
Studies  PI  and  P2  this  test  was  the  Inventory  of  Personal  Attitudes,  and 
on  Study  P3  the  Edwards  Personal  Preference  Schedule  was  used.  On  the 
first  two  of  these  panel  studies  the  results  of  these  tests  were  used  as  a  gen- 
eral guide,  primarily  to  detect  applicants  with  extreme  personality  traits. 
However,  by  the  time  of  Study  P3,  evidence  had  been  obtained  that  inter- 
viewers who  scored  high  on  "dominance"  and  low  on  "deference"  ap- 
peared to  be  more  successful  than  other  interviewers  in  obtaining  complete 
financial  data.141  As  a  result,  these  personality  characteristics  were  used 
as  a  primary  basis  for  selecting  interviewers  for  Study  P3.  Additional  evi- 
dence accumulated  on  that  study  tended  to  reinforce  the  earlier  findings, 
although  such  evidence  can  only  be  circumstantial  in  view  of  the  non- 
experimental  nature  of  the  interviewer  selection  process. 

Nevertheless,  these  results,  supported  by  a  multivariate  analysis,  sug- 
gest that  personality  tests  may  serve  as  a  useful  device  for  selecting 
interviewers.  In  particular,  these  tests  apparently  pinpoint  certain  char- 
acteristics of  interviewers  not  commonly  suspected.  For  example,  there 
was  some  tendency  for  the  completeness  of  information  to  be  inversely 
correlated  with  the  overall  response  rates  obtained  by  individual  inter- 
viewers.142 

Selection  Procedure 

Experience  with  the  various  survey  operations  led  to  the  development 
of  a  two-stage  procedure  for  selecting  interviewers.  The  first  stage  in- 
volved the  selection  of  so-called  "interviewer  candidates"  from  among  the 
applicants  for  interviewer  positions.  These  interviewer  candidates  were 
chosen  by  means  of  a  scoring  system  based  on  the  completeness  with  which 
the  application  form  was  filled  out,  on  a  checklist  filled  out  by  a  project 
staff  member  after  interviewing  the  applicant,  on  the  scores  of  the  person- 
ality tests,  and  on  the  quality  of  the  reference  letters.  The  scoring  system 
was  such  that  twice  as  much  emphasis  was  given  to  the  checklist  and 
to  the  evaluation  of  the  reference  letters  as  to  either  of  the  two  other 
criteria.143 

As  a  rule,  the  procedure  for  selecting  interviewer  candidates  consisted 
of  ordering  applicants  by  their  total  score  and  selecting  the  requisite  num- 
ber with  the  highest  scores.  However,  certain  additional  factors  were 
taken  into  consideration.  One  such  factor  was  the  location  of  the  appli- 
cant relative  to  the  concentration  of  the  sample  members.  A  second  factor 
was  the  applicant's  interviewing  experience,  which,  as  a  rule,  was  not  con- 

141  Ibid.,  Chapter  III. 

142  Ibid. 

143  For  the  exact  scoring  method,  see  ibid.,  p.  46. 
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sidered  a  point  in  his  favor.  This  was  particularly  true  if  the  applicant 
had  done,  say,  primarily  attitudinal  interviewing  of  an  unstructured  type 
when  interviewers  were  to  be  trained  to  follow  a  highly  structured  ap- 
proach. 

After  interviewer  candidates  had  been  selected,  they  were  put  through 
a  series  of  intensive  training  sessions.  In  the  course  of  these  sessions,  dis- 
cussed briefly  in  the  following  section,144  candidates  were  observed  while 
practice-interviewing  and  were  given  qualifying  take-home  examinations 
that  covered  financial  terminology  as  well  as  interviewing  procedures.  In 
addition,  candidates  were  assigned  trial  interviews  with  predesignated  re- 
spondents. In  one  study,  these  were  actual  "dummy"  interviews,  in  the 
sense  that  the  respondents  had  already  been  approached  by  the  project 
staff  and  had  been  coached  on  how  to  act  and  on  what  information  to 
report.  Such  interviews  proved  to  be  highly  valuable,  since  the  interviewer 
report  could  be  compared  with  the  respondent  report  obtained  after  the 
interview.  Candidates  that  survived  this  procedure  were,  with  few  excep- 
tions, hired  as  project  interviewers. 

Although  better  procedures  might  have  been  devised,  this  procedure 
appears  to  have  been  fairly  effective  in  selecting  project  interviewers.  It 
was  a  fairly  expensive  procedure,  since,  as  a  rule,  nine  applicants  had  to 
be  hired  to  produce  one  project  interviewer. 

Training 

Training  Sessions 

Training  sessions  on  technical  studies  have  to  be  of  two  types.  First, 
training  is  necessary  at  the  outset  to  acquaint  the  interviewers  with  the 
objectives  and  reasons  for  the  study  as  well  as  with  how  to  conduct  the 
interview  and  fill  out  the  questionnaires.  Second,  if  the  study  is  a  panel 
operation,  additional  training  sessions  are  required  to  instruct  the  inter- 
viewers on  methods  and  procedures  in  each  following  wave  of  interviews. 
Although  no  formal  experiments  were  carried  out  with  either  type  of 
training  session,  the  experience  with  these  sessions  suggests  preferred  ways 
of  handling  them. 

Since  the  interviewer  candidates  at  the  initial  training  session  were 
largely  unacquainted  with  the  general  subject  of  consumer  finances,  con- 
siderable emphasis  on  this  subject  was  required.  If  the  interviewers  could 
be  made  to  understand  the  value  of  consumer  financial  data  and  the  uses 
to  which  they  might  be  put,  there  was  a  much  greater  likelihood  that  they 
would  remain  with  the  study  and  do  a  conscientious  job  of  collecting  data. 
Hence,  the  latter  part  of  an  initial  two-to-three-hour  session  was  devoted 

144  For  further  details,  see  ibid.,  Chapter  IV. 
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to  coverage  of  the  subject  matter  of  the  study,  especially  to  the  necessity  of 
obtaining  complete  and  accurate  financial  data.  Later  meetings  in  the 
initial  training  sessions  were  devoted  to  a  review  of  interviewing  tech- 
niques, to  means  of  dealing  with  the  respondents,  and  to  procedures  for 
dealing  with  difficult  situations.  Only  after  this  material  had  been  covered 
did  the  sessions  focus  on  the  particular  questionnaires  to  be  used. 

Experience  with  the  initial  pilot  study  in  Chicago  indicated  a  need 
for  additional  training  sessions  before  each  new  wave  of  interviews.  Hence, 
two-to-three-hour  training  sessions  were  held  in  advance  of  each  wave  of 
interviews  on  all  of  the  later  studies.  These  sessions  proved  useful,  not 
only  for  instructing  interviewers  in  the  particular  forms  and  procedures 
to  be  used,  but  also  in  correcting  weak  spots  in  interviewer  practices  and 
in  reinforcing  the  interviewer's  knowledge  of  the  purpose  and  objectives 
of  the  study.  In  addition,  in  Study  P3,  interviewers  on  later  waves  were 
required  to  fill  in  a  copy  of  the  questionnaire  on  the  basis  of  information 
supplied  by  the  field  director  at  the  beginning  of  the  training  session,  and 
these  questionnaires  were  collected  and  checked  for  accuracy. 

Value  of  Qualifying  Examination 

As  was  mentioned  earlier,  interviewer  candidates  were  given  a  take- 
home  qualifying  examination  as  part  of  the  initial  training  sessions.  This 
examination  covered  interviewing  methods  and  procedures,  the  purposes 
and  objectives  of  the  study,  and  the  meaning  of  numerous  financial  terms. 
The  examination  was  detailed  enough  that  training  manuals  and  other 
written  materials  had  to  be  consulted  in  order  to  pass.  In  this  way,  it 
was  felt  that  the  interviewer  candidates  would  become  better  acquainted 
with  the  details  of  the  study.  A  passing  grade  was  required  before  an 
interviewer  candidate  could  be  hired  as  an  interviewer,  although  candi- 
dates with  higher  scores  were  not  necessarily  given  preference  over  those 
with  lower  scores. 

There  is  little  doubt  that  these  examinations  made  interviewers  better 
acquainted  with  the  study  and  with  the  interviewing  procedures.  The  ex- 
amination also  brought  out  aspects  of  the  study  which  had  been  mis- 
understood and  which  could  be  clarified  before  actual  interviewing  began. 
Moreover,  the  complexity  and  time  required  for  the  examination  to  be 
completed  (at  least  two  hours)  served  in  each  study  to  weed  out  several 
candidates  who  presumably  were  not  sufficiently  interested  to  expend  the 
effort  to  finish  the  examination.  For  these  reasons  alone,  an  examination 
of  this  type  would  seem  to  be  a  useful  training  tool. 

Whether  or  not  such  an  examination  is  even  more  effective  and  helps 
to  differentiate  good  from  poor  interviewers  is,  however,  not  clear.  In 
terms  of  interviewer  dropouts,  the  examination  does  not  support  this  view, 
to  judge  by  comparison  between  the  average  scores  of  those  interviewers 
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who  served  on  all  five  waves  of  the  study  and  of  those  who  dropped  out 

before  the  study  was  over. 

Average  score 

Study 

P2 

P3 


Dropouts 

Others 

128 

129 

113 

114 

In  terms  of  interviewer  efficiency,  the  evidence  is  mixed,  as  is  shown 
by  correlations  computed  for  Studies  P2  and  P3  between,  first,  the  inter- 
viewer score  on  this  examination  and  his  response  rate,  and  second,  be- 
tween the  score  and  his  pick-up  of  the  validated  holding.  None  of  these 
correlations  is  statistically  significant  at  the  .05  level. 

Correlation  coefficient  between 
examination  score  and 


Study  Response  rate         Pickup  rate 

P2 -.40  -.01 

P3 15  .05 

In  view  of  the  relatively  small  sample  sizes  and  lack  of  randomization, 
it  is  difficult  to  draw  firm  inferences  from  these  data.  However,  at  the 
least,  these  results  suggest  that  such  an  examination  is  a  useful  part  of  the 
training  sessions. 

Supervision 

Interviewer  supervision  has  many  facets,  ranging  from  the  method  of 
compensation  to  the  processing  of  the  completed  questionnaires.  Those 
facets  which  received  special  attention  are  discussed  in  this  section. 

Compensation 

On  the  initial  pilot  study  and  on  Study  PI,  interviewers  were  paid  on 
an  hourly  basis,  portal  to  portal,  with  expenses  for  mileage,  phone  calls, 
and  miscellaneous  items.  In  an  effort  to  attract  and  retain  good  inter- 
viewers, the  rate  offered  was  set  above  the  going  hourly  rate  at  the  time. 
In  addition,  as  an  incentive  to  remain  with  the  study,  the  rate  of  compen- 
sation was  raised  five  cents  per  hour  on  each  following  wave. 

Nevertheless,  various  shortcomings  became  apparent  with  this  method 
of  compensation.  The  interviewers,  who  were  largely  professional  people, 
seemed  dissatisfied  with  an  hourly  wage.  Perhaps  for  this  reason,  assign- 
ments were  not  often  completed  on  schedule.  Also,  high  variability  in  cost 
per  interview  was  apparent  from  one  interviewer  to  another,  a  variability 
which  was  larger  among  interviewers  than  among  respondents  for  the 
same  interviewer.  The  average  cost  per  interview  was  also  not  low,  and 
there  was  reason  to  suspect  that  some  interviewers  were  unintentionally 


236     The  Reliability  of  Consumer  Reports  of  Financial  Assets  and  Debts 

inflating  their  expenses  by  visiting  noncontacts  at  times  when  they  did  not 
really  expect  them  to  be  home. 

As  a  result,  the  method  of  compensation  was  altered  on  Study  P2,  and 
interviewers  were  paid  by  the  completed  interview  instead  of  by  the 
hour.  Based  on  cost  figures  from  the  previous  studies,  $10  was  paid  for 
each  of  the  first  10  completed  interviews,  $13  for  all  interviews  over  10, 
and  $15  for  interviews  with  panel  members  previously  refusing  another 
interviewer.  Once  established,  these  higher  rates  remained  the  same  for 
all  future  waves,  regardless  of  the  interviewer,  with  the  exception  that 
refusal  of  an  interview  by  a  previously  cooperative  panel  member  moved 
him  into  the  $15  bracket  if  the  refusal  were  converted.  Supplementing 
these  interview  rates  was  an  hourly  rate  of  payment  for  attending  training 
sessions  and  for  study  time  and  a  mileage  reimbursement  for  panel  mem- 
bers living  more  than  10  miles  from  the  interviewer's  home. 

A  similar  method  of  compensation  was  employed  on  Study  P3  with 
two  modifications.  First,  the  rate  per  interview  was  established  at  $10  for 
the  first  10  interviews  on  each  wave  and  $13  for  all  interviews  over  10  — 
to  give  interviewers  some  incentive  to  obtain  more  than  10  interviews.  In 
addition,  interviewers  were  paid  excess  mileage  on  all  assignments  over  10 
miles  from  their  home  regardless  of  the  outcome  —  whether  it  was  an 
interview,  a  refusal,  or  a  noncontact.  Such  reimbursements  seemed  espe- 
cially proper  on  that  particular  study  because  of  its  farm  location  and  the 
large  distances  interviewers  had  to  travel. 

All  things  considered,  payment  by  the  completed  interview  appeared 
to  be  preferable  to  payment  by  the  hour.  Interviewer  morale  appeared  to 
be  higher,  perhaps  because  the  method  of  payment  was  more  in  agreement 
with  the  method  by  which  these  (professional)  people  were  paid  on  their 
regular  jobs.  Moreover,  surprisingly  enough,  compensation  by  the  inter- 
view was,  if  anything,  less  expensive  than  compensation  by  the  hour,  as 
is  shown  by  the  following  tabulation  (these  figures  include  compensation 
for  meetings,  mileage  expenses,  and  trips  by  project  staff  to  the  field)  : 

Average  cost  per  interview 

Wave  Pilot  study  Study  PI  Study  P2  Study  P3 

1 $18. 87145  $23.11  $16.87  $14.79 

2 18.46  20.61  13.62  13.15 

3 18.23  15.42  13.23  14.35 

4 18.19  14.41  12.58  12.49 

5 19.02  10.03  13.68  13.92 

Average 18.55  16.72  14.00  13.74 

145  This  figure  is  relatively  low  because  this  panel  operation  had  been  pre- 
ceded by  a  so-called  "background  interview"  with  the  same  panel  members  about 
six  months  earlier,  so  that  much  of  the  original  contact  work  was  out  of  the  way 
by  the  first  wave. 
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The  cost  figures  shown  in  the  pilot  study  are  not  directly  comparable 
with  the  others  because  most  of  the  interviewing  on  that  study  was  sub- 
contracted. Nevertheless,  it  is  clear  that  the  average  cost  per  interview 
on  Studies  P2  and  P3,  the  latter  with  the  extra  mileage  expenses,  was  still 
substantially  less  than  the  average  cost  per  interview  on  Study  PI.  In 
addition,  contrary  to  what  was  feared,  there  was  no  evidence  that  inter- 
viewers were  trying  to  minimize  the  amount  of  time  spent  by  rushing  the 
respondents.  On  both  of  these  studies,  an  interviewer  did  not  receive  any 
compensation  if  a  questionnaire  was  turned  in  with  many  blanks. 

Interviewer  Rotation 

A  personal  interview  panel  raises  the  danger  that  the  interviewers  and 
the  respondents  may  become  so  well  acquainted  that  the  later  interviews 
are  regarded  as  mere  formalities.  This  is  especially  true  of  those  panels  in 
which  the  main  objective  of  the  later  interviews  is  to  update  previously 
obtained  information. 

One  means  of  coping  with  this  difficulty  is  to  rotate  the  interviewers 
among  the  respondents  on  later  waves.  Such  a  procedure  is  not  popular 
with  interviewers,  however,  because  it  disrupts  previously  established  rela- 
tionships. Moreover,  according  to  the  arguments  of  the  interviewers,  such 
a  procedure  can  increase  the  refusal  rate  because  contact  with  a  new  in- 
terviewer may  irritate  the  respondent  and  lead  him  to  refuse  to  cooperate 
any  further.  Presumably,  this  would  be  especially  so  if  the  respondent  had 
been  asked  to  supply  confidential  information  and  if  he  were  then  asked 
to  supply  the  same  information  to  a  new  interviewer. 

On  the  other  hand,  interviewer  rotation  may  have  advantages  because 
of  the  very  disruption  of  previously  established  relationships.  Thus,  a  new 
interviewer  might  not  take  previously  listed  holdings  so  much  for  granted 
as  the  old  interviewer  and,  in  pressing  for  additional  information,  might 
uncover  holdings  hitherto  not  reported. 

To  test  these  ideas,  a  moderate  amount  of  interview  rotation  was  car- 
ried out  on  Wave  3  of  both  Studies  P2  and  P3.  On  Study  P2,  three  re- 
spondents were  taken  away  from  each  interviewer  and  three  were  assigned 
to  him  from  other  interviewers.  In  these  assignments,  an  attempt  was 
made  to  hold  constant  the  type  of  form  handled  by  each  interviewer 
(change  versus  holdings),  the  approximate  overall  travel  distance,  and 
the  rate  per  interview.  Sample  members  who  had  refused  to  supply  all 
financial  data  were  not  included,  although  this  did  not  prevent  sample 
members  who  had  not  reported  validated  holdings  from  being  included. 
Subject  to  these  restrictions,  interviews  were  rotated  on  a  random  basis. 

The  rotation  procedure  on  Study  P3  was  essentially  the  same,  with 
three  to  five  sample  members  rotated  for  each  interviewer.   As  before,  the 
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Nonrotated 

94% 

98% 

91 

94 
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type  of  form  was  held  constant,  as  was  the  travel  distance  assigned  to  each 
interviewer.  In  this  study,  however,  an  effort  was  made  to  rotate  more 
panel  members  who  had  not  supplied  validated  data,  because  the  results 
of  the  rotation  on  Study  P2  suggested  that  pickup  of  nonreported  holdings 
was  increased  in  this  manner. 

The  results  of  these  experiments  do  not  provide  much  support  for  the 
desirability  of  interviewer  rotation.  The  response  rates  obtained  on  the 
rotated  and  the  nonrotated  segments  of  the  samples  on  each  of  the  two 
studies,  shown  in  the  following  tabulation,  fail  to  bring  out  any  superiority 
for  the  rotated  interviews.  If  anything,  a  slightly  higher  proportion  of  the 
sample  members  contacted  were  interviewed  on  the  nonrotated  segment. 

Response  rate 

Study 

P2 

P3 

The  results  relating  to  the  pickup  rate  are  similar.  As  with  the  re- 
sponse rate,  the  proportion  of  previously  nonreported  validated  holdings 
picked  up  on  the  third  interview  was  slightly  higher  in  each  study  among 
the  nonrotated  sample  members  than  among  the  rotated  sample  members. 

Pickup  rate 

Study 

P2 

P3 

At  the  same  time,  these  results  do  not  preclude  the  possibility  that  the 
rotation  procedure  may  have  produced  intangible  benefits  reflected  in 
both  segments  of  the  sample.  Thus,  the  fact  that  interviewers  were  in- 
formed at  the  beginning  of  the  study  of  this  rotation  plan  (but  not  for 
which  interviews)  may  have  influenced  them  to  push  harder  on  each 
interview 'for  complete  listing  of  holdings  than  might  otherwise  have  been 
the  case. 

Quality  Control 

Virtually  all  of  the  standard  techniques  were  used  to  maintain  inter- 
viewer morale  and  to  ensure  that  questionnaires  were  filled  out  in  accord- 
ance with  prior  specifications.  These  included  periodic  group  meetings 
(mentioned  earlier),  meetings  with  individual  interviewers  to  clarify  par- 
ticular aspects  of  the  study,  mailing  of  so-called  "interim  reports"  to  both 
panel  members  and  interviewers,  and  editing  of  questionnaires  both  in  the 
field  and  at  project  headquarters.  Field  supervisors  were  hired  for  all  of 
the  studies  but  Study  P3  in  which,  because  of  the  proximity  of  the  office 
to  the  sample  area,  the  project  field  director  acted  as  the  field  supervisor. 
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In  that  particular  study  all  editing  was  done  at  the  office  but,  again  be- 
cause of  the  same  proximity,  interviewers  received  edited  questionnaires 
needing  further  clarification  at  least  as  quickly  as  on  any  previous  studies. 

Besides  these  methods,  four  additional  procedures  were  used  at  various 
times  to  maintain  and  improve  the  quality  of  the  information.  First,  inter- 
viewers were  required  to  complete  an  Interviewer  Report  Form  for  every 
contact  attempt,  whether  successful  or  not;  one  such  form  is  shown  in 
Appendix  A.  The  specific  content  of  this  form  varied  with  the  particular 
wave  and  study,  but  in  all  instances  an  attempt  was  made  to  bring  out  on 
this  form  the  circumstances  and  results  of  the  contact  as  well  as  the  inter- 
viewer's interpretation  of  the  quality  of  the  data.  Interviewers  were  also 
asked  to  record  on  this  form  any  peculiarities  of  the  interview  or  circum- 
stances which  might  reflect  on  the  accuracy  or  completeness  of  the  finan- 
cial data  or  on  the  cooperativeness  of  the  respondent.  As  was  noted  in 
Chapters  III-V,  these  interviewer  ratings  were  correlated  to  some  extent 
with  the  reliability  of  the  data,  judging  by  the  reports  on  the  validated 
holdings.  Evidence  also  exists  to  support  the  hypothesis  that  the  value  of 
these  ratings  tends  to  increase  with  the  quality  of  the  interviewer.146  Aside 
from  these  indications,  the  Interviewer  Report  Form  undoubtedly  served 
to  increase  the  emphasis  on  securing  accurate  and  complete  information. 

A  second  technique  was  the  use  of  telephone  calls  and  return  postcards 
to  check  on  the  interview  and  on  the  performance  of  the  interviewer. 
Although  interviewer  cheating  is  not  a  major  problem  in  a  panel  opera- 
tion, such  a  technique  may  serve  to  detect  cheating  when  it  occurs,  and  it 
did  serve  to  bring  to  light  two  such  instances  in  Study  P2.  On  the  other 
hand,  the  use  of  postcards  alone  to  detect  cheating  may  not  be  very  effi- 
cient, to  judge  by  the  one  experiment  carried  out  in  Chicago.  In  this 
experiment,  10  people  selected  at  random  from  the  Chicago  telephone 
directory  were  sent  thank-you  letters  on  University  of  Illinois  stationery 
for  granting  an  interview  on  their  finances.  Although  these  people  were 
not  actually  interviewed,  five  returned  the  postcard  indicating  the  length 
of  the  interview  and  their  opinion  of  the  interviewer! 

The  telephone  calls  and  the  return  postcards  were  useful  primarily  in 
detecting  interviewer  qualities  which  tended  to  irritate  respondents.  Such 
defects  were  brought  to  the  interviewer's  attention  without  indicating 
which  respondents  were  involved.  Since  the  interviewers  were  told  that 
these  postcard  checks  and  telephone  calls  were  being  made  continuously, 
they  were  under  pressure  to  correct  these  defects  (and  interviewer  cheat- 
ing was  also  forestalled) . 

The  third  technique  was  to  impose  deadline  dates  for  individual  inter- 
viewees rather  than  for  an  interviewer's  complete  assignment.    Individual 

146  Ibid.,  Chapter  III. 
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questionnaire  deadline  dates  were  imposed  on  the  last  two  waves  of 
Study  P3,  and  they  were  highly  successful  in  ensuring  that  the  entire  wave 
of  interviews  would  be  completed  on  schedule.  This  technique  also  served 
to  counteract  the  tendency  of  interviewers  to  leave  their  least-cooperative 
respondents  for  the  very  end.  In  practice,  interviewers  were  given  a  dead- 
line of  returning  the  completed  questionnaire  within  one  week  after  the 
respondent  received  the  advance  letter.  Less  than  10  interviews  were  left 
outstanding  with  an  interviewer  at  any  particular  time,  so  that  he  would 
not  feel  rushed  in  making  these  interviews. 

The  use  of  validation  data  for  pinpointing  poor  interviewers  served  as 
a  fourth  means  of  quality  control,  although  its  full  potential  may  not  have 
been  realized  in  these  studies.  With  the  aid  of  the  validation  data,  inter- 
viewers who  were  picking  up  a  much  smaller  proportion  of  the  validated 
holdings  than  other  interviewers  could  be  singled  out,  and  the  circum- 
stances of  these  interviews  could  be  investigated.  Such  data  can  provide 
conclusive  evidence  of  interviewer  cheating,  as  was  true  of  the  two  in- 
stances in  Study  P2  mentioned  earlier.  Even  more  important,  such  data 
indicate  which  interviewers  may  not  be  obtaining  complete  financial  data 
and  for  which  respondents.  Given  such  knowledge,  these  interviewers  may 
be  trained  more  thoroughly  on  means  of  soliciting  financial  data  or,  if 
these  attempts  are  unsuccessful,  the  interviews  might  be  reassigned  to 
interviewers  more  successful  in  obtaining  these  data.  Apart  from  the 
planned  rotation  of  interviewers,  such  attempts  were  not  made  in  these 
studies  because  they  would  have  interfered  with  the  experimental  design. 
Nevertheless,  the  use  of  validation  data  to  control  interview  quality  would 
seem  to  offer  considerable  promise. 

Other  Controls 

A  variety  of  other  methods  was  used  to  improve  data  reliability  both 
initially  and  during  the  later  waves  of  the  panel  operations.  These  meth- 
ods varied  from  the  use  of  respondent  incentives  to  attempting  a  more 
narrowly  focused  definition  of  the  sampling  unit. 

Definition  of  the  Sampling  Unit 

Lack  of  knowledge  by  the  respondent (s)  of  the  holdings  of  different 
family  members  may  be  a  major  source  of  response  error,  even  when  the 
"savings  unit"  definition  is  employed.  There  is  little  doubt  that  errors  due 
to  lack  of  knowledge  would  have  been  even  more  substantial  if  the  more 
customary  definition  of  sampling  unit  had  been  employed,  such  as  the 
"consumer  unit"  or  the  "family."  How  much  error  was  the  result  of  lack 
of  knowledge  in  such  instances  can  only  be  conjectured. 
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To  minimize  errors  due  to  lack  of  knowledge,  a  narrow  definition  of 
the  sampling  unit  was  formulated  at  the  outset  and  was  used  on  all  of  the 
panel  studies.  Illustrative  of  this  definition  is  the  one  used  in  Study  P3, 
namely,  the  farmer,  "your  wife,  your  children  under  16  years  of  age,  and 
other  family  members  who  earn  less  than  $600  per  year  and  who  have  less 
than  $600  in  their  own  name." 

A  definition  of  this  type  had  at  least  two  advantages.  First,  it  served 
to  focus  coverage  on  the  immediate  family  of  the  farmer,  particularly  on 
those  still  financially  dependent  on  the  head  of  the  household.  By  restrict- 
ing asset  and  debt  coverage  to  this  group,  omissions  due  to  lack  of  knowl- 
edge appear  to  have  been  minimized  although  not  completely  eliminated. 
Second,  the  definition  is  relatively  simple  and  was  readily  understood  by 
virtually  all  the  respondents,  to  judge  by  the  few  questions  of  interpreta- 
tion raised  in  the  interviews. 

On  the  other  hand,  such  a  definition  does  present  the  disadvantage  of 
dividing  the  family  or  the  household  into  several  components,  which  may 
present  problems  of  both  economic  and  statistical  analysis.  Such  problems 
are  not  unsurmountable,  however,  and  could  readily  be  handled  with  ap- 
propriate transformations  of  the  data.  For  example,  allowance  could  be 
made  in  the  sampling  frame  for  the  fact  that  each  household  may  contain 
more  than  one  savings  unit.  In  each  household  where  a  savings  unit  is 
interviewed,  pertinent  information  could  be  obtained  on  the  number  and 
selected  characteristics  of  other  savings  units  in  the  same  household.  In 
this  way,  information  obtained  from  a  sample  of  savings  units  could  be 
transformed  into  estimates  pertaining  to  families  or  any  other  consumer 
unit. 

Motivating  the  Respondent 

Various  attempts  were  made  to  involve  the  respondent  more  closely 
with  the  study.  One  approach  was  to  use  the  advance  letter  to  respon- 
dents for  this  purpose.  On  all  three  panel  studies,  PI  to  P3,  the  covering 
letter  to  a  split-run  half  of  the  sample  on  the  first  wave,  in  addition  to  the 
usual  request  for  cooperation,  asked  the  respondent  to  evaluate  the  inter- 
view procedures  at  the  end  of  the  interview  and  to  offer  suggestions  for 
improvement.  When  the  interview  was  over,  these  sample  members  were 
left  a  form  to  be  returned  by  mail,  on  which  to  record  their  impressions 
of  the  interview  and  to  make  suggestions  for  changes. 

The  response  rate  on  all  three  studies  was  only  barely  higher  among 
those  receiving  this  special  letter  than  among  the  other  sample  members. 
However,  over  61  percent  of  the  forms  requesting  suggestions  were  re- 
turned and  about  one-half  of  the  returned  forms  contained  written  com- 
ments or  suggestions  regarding  the  interview  procedure.    Moreover,  this 
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technique  may  have  increased  the  reliability  of  the  data,  as  is  evident  from 
the  following  tabulation  showing,  acccording  to  the  type  of  advance  letter 
received,  the  percentage  of  interviewed  respondents  that  reported  the 
validated  holding. 

Study  Regular  letter  Special  letter 

P2  (time  deposits) 59%  65% 

P3  (debt) 44  45 

P3  (time  deposits) 8  12 

Also,  those  originally  receiving  the  evaluation  letter  were  more  likely 
to  remain  in  the  panel  on  later  waves  on  both  studies,  although  the  differ- 
ences are  not  statistically  significant.  Since  use  of  this  evaluation  approach 
entails  little  extra  cost,  it  would  seem  to  be  a  worthwhile  procedure. 

Use  of  References 

A  characteristic  incorporated  in  all  of  the  advance  letters  on  the  three 
panel  studies,  PI  to  P3,  was  the  mention  of  names  of  prominent  local 
people  whom  the  sample  members  could  contact  to  verify  the  authenticity 
of  the  study.  As  a  rule,  these  people  were  well-known  and  highly  regarded 
top  administrators  in  local  colleges  and  universities. 

Since  this  technique  was  not  part  of  a  controlled  experiment,  its  exact 
effect  on  response  cannot  be  measured.  Nevertheless,  there  is  no  question- 
ing that  the  effect  was  positive,  since  no  instance  was  reported  of  a  sample 
member  refusing  to  cooperate  because  of  reference  to  these  people  in  the 
advance  letter.  At  the  same  time,  the  interviewers  reported  numerous 
instances  where  sample  members  cooperated  only  after  contacting  one  of 
the  people  mentioned  in  the  advance  letter,  and  these  instances  were  veri- 
fied by  the  references  themselves.  In  addition,  other  sample  members  indi- 
cated that  the  mention  of  one  of  these  person's  names  in  the  advance  letter 
influenced  them  to  cooperate  without  actually  calling  the  individual. 

Gifts  and  Incentives 

Various  attempts  were  made  to  induce  respondents  to  cooperate  by 
offering  them  gifts  and  other  incentives.  On  the  first  interview  of  the  pilot 
panel  operation,  a  choice  of  several  gifts  was  offered  to  the  respondent,  at 
the  discretion  of  the  interviewer.  According  to  reports  received  from  the 
interviewers,  the  offer  of  a  gift  contributed  noticeably  to  securing  coopera- 
tion from  about  one-fifth  of  those  interviewed.147  In  some  instances  inter- 
viewers also  reported  that  the  gifts  influenced  respondents  to  be  more 
cooperative  on  subsequent  interviews. 

147  Ferber,  Collecting  Financial  Data  by  Consumer  Panel  Techniques,  op.  cit., 
pp.  26-27. 
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At  the  beginning  of  the  fourth  wave  of  the  same  study,  panel  members 
were  sent  as  a  surprise  gift  a  box  of  assorted  cheeses  and,  somewhat  later, 
a  copy  of  a  press  release  about  the  project.  Due  either  to  the  cheese  or  to 
the  press  release,  a  clear  improvement  in  cooperativeness  was  detected  on 
that  wave,  as  measured  by  interviewer  ratings  and  by  the  frequency  of 
data  refusals.148 

On  the  other  hand,  offering  cash  to  respondents  was  not  at  all  success- 
ful in  Study  S5.  In  that  study,  half  of  the  sample  members  were  offered 
a  $10  payment  by  the  interviewer.  This  offer  appears  to  have  boomer- 
anged,  since  the  response  rate  was  actually  lower  among  the  paid  group, 
with  no  appreciable  reduction  in  response  error.149  Whether  it  was  due  to 
objections  to  a  university  offering  money  to  the  public  or  whether  it  was 
due  to  some  other  factor  is  not  clear,  but  on  future  operations  of  the 
project  no  attempt  was  made  to  pay  respondents. 

On  later  studies,  sample  members  were  sent  copies  of  interim  reports 
and  other  project  publications  (excluding  those  relating  to  validation). 
On  the  basis  of  observation,  the  most  effective  of  all  appears  to  have  been 
local  newspaper  releases.  In  various  instances,  photostats  were  prepared 
of  these  releases  and  sent  to  the  sample  members.  Judging  by  interviewer 
reports,  these  newspaper  stories  were  useful  both  in  convincing  sample 
members  of  the  authenticity  and  value  of  the  study  and  in  maintaining 
their  interest. 

Attitudinal  Questions 

Attitudinal  questions  were  used  extensively  in  the  initial  studies  of  the 
project,  primarily  at  the  beginning  of  the  questionnaire  to  "soften  up"  the 
respondent.  This  approach  proved  effective  in  many  instances,  but  occa- 
sionally it  was  almost  too  effective. 

Interviewers  appeared  to  be  spending  so  much  time  on  the  attitudinal 
questions  that  by  the  time  requests  were  made  for  dollar  balances  and 
names  of  institutions,  respondents  were  somewhat  tired  and  were  reluctant 
to  keep  on  cooperating.  As  a  result,  two  modifications  were  made  in  later 
questionnaires,  in  Studies  P2  and  P3.  In  Study  P2,  the  attitudinal  ques- 
tions were  interspersed  among  the  requests  for  data,  partly  to  cope  with 
the  aforementioned  difficulty  and  partly  to  relieve  the  monotony  of  con- 
tinual requests  for  data.  To  judge  by  interviewer  reports,  this  technique 
was  fairly  successful,  although  no  clear-cut  evidence  was  obtained  that  its 
use  produced  information  that  would  not  otherwise  have  been  obtained. 

As  a  result,  a  second  expedient  was  tried  of  reducing  drastically  the 
number  of  attitudinal  questions.  This  procedure  was  followed  in  Study  P3, 

148  Ibid.,  pp.  27-30. 

149  Lansing,  Ginsburg,  and  Braaten,  op.  cit.,  Chapter  VII. 
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on  the  first  wave  of  which  virtually  no  attitudinal  information  was  re- 
quested. On  the  basis  of  observation,  this  procedure  appears  to  have  been 
successful,  and  very  good  cooperation  was  obtained  on  this  wave  (al- 
though, as  was  noted  previously,  many  other  factors  were  also  involved) . 
Later  waves  of  Study  P3,  consequently,  also  sought  very  little  attitudinal 
information.  Such  information  would  seem  worth  requesting  only  if  de- 
sired for  its  own  sake  rather  than  to  serve  as  a  prelude  for  obtaining  finan- 
cial data. 

Data  Reliability 

Regardless  of  which  methods  are  used  to  collect  financial  data  in  a 
consumer  survey,  some  means  of  evaluating  the  reliability  of  these  data 
will  be  desired.  The  methods  employed  for  this  purpose  in  the  present 
studies  have  been  discussed  in  earlier  chapters  and  are  therefore  covered 
here  only  briefly. 

Interviewer  Ratings 

The  use  of  interviewer  ratings  to  gauge  the  reliability  of  the  data  has 
been  covered  to  some  extent  in  Chapter  VI,  in  which  a  definite  correlation 
was  shown  to  exist  between  these  ratings  and  the  tendency  of  sample 
members  to  remain  in  the  panel.  These  ratings  were  also  shown  to  be 
related  in  varying  degrees  to  both  the  reporting  of  the  validated  holding 
and  to  the  accuracy  of  this  report.  The  nature  of  the  correlations  was 
sometimes  unexpected,  however,  as  exemplified  by  the  results  shown  in 
Chapter  III  where  a  higher  correlation  was  obtained  between  the  ac- 
curacy rating  and  nonreporting  than  that  obtained  between  the  complete- 
ness rating  and  nonreporting.  There  is  some  evidence  that  these  ratings 
tended  to  be  more  reliable  for  interviewers  who  were  most  successful  in 
picking  up  the  validated  debt.  Presumably,  the  more  effective  interviewers 
were  also  more  sensitive  to  the  respondent's  reaction  to  the  request  for 
financial  data  and  were  better  able  to  gauge  the  true  situation. 

Evidence  from  other  studies  provides  mixed  support  for  the  value  of 
these  ratings.  Thus,  in  Study  S2,  validating  car  debt,  interviewer  judg- 
ments of  the  accuracy  of  response  bore  no  relation  to  the  true  accuracy.150 
On  the  other  hand,  in  Study  S3,  validating  savings  accounts,  "The  inter- 
viewers were  most  successful  in  telling  whether  the  respondents  failed  to 
report  the  account  completely."151 

The  value  of  interviewer  ratings  for  detecting  different  forms  of  re- 
sponse errors  is  brought  out  in  Table  81,  which  shows  the  proportions  of 
members  of  different  panels  reporting  validated   holdings  by   different 

150  Ibid.,  Chapter  III. 

151  Ibid.,  p.  185.  These  results  may  have  been  affected  by  the  possibility  that 
some  interviewers  may  have  suspected  that  the  particular  sample  contained  only 
owners  of  savings  accounts. 
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ratings  of  respondent  cooperativeness.  As  is  evident  from  this  table,  the 
frequency  and  accuracy  of  reporting  of  the  validated  holding  for  life 
insurance  and  savings  accounts  tend  to  increase  sharply  with  the  favor- 
ableness  of  the  interviewer  rating  on  several  characteristics,  although  not 
so  for  debt.  The  accuracy  and  completeness  ratings  for  the  specific  assets 
appear  to  be  especially  useful  in  this  regard. 

Use  of  Records 

In  some  ways,  information  on  the  use  of  records  by  respondents  is  a 
more  practicable  indicator  of  data  reliability  than  interviewer  ratings, 
because  of  the  greater  objectivity  of  the  former.  The  value  of  this  indi- 
cator has  been  borne  out  in  previous  chapters  where  it  was  shown  that  for 
many  of  the  validated  holdings  both  completeness  and  accuracy  of  report 
were  correlated  positively  with  the  use  of  records.  Moreover,  like  inter- 
viewer ratings,  use  of  records  tends  to  reflect  the  propensity  of  sample 
members  to  remain  in  the  panel  (see  p.  165) . 

Nevertheless,  the  full  potentialities  of  a  use-of-records  variable  remains 
to  be  explored.  No  attempt  was  made  in  these  past  studies  to  ascertain 
exactly  what  types  of  records  were  consulted  nor  to  relate  accuracy  and 
completeness  of  report  to  these  different  types  of  records.  Stratification  of 
use  of  records  in  such  ways  may  well  improve  the  effectiveness  of  this 
indicator  of  data  reliability. 

Respondent  Evaluation 

A  different  and  possibly  novel  approach  is  to  request  the  respondent  to 
provide  an  estimate  of  the  accuracy  of  the  data  that  he  is  supplying.  This 
approach  was  tested  in  Studies  P2  and  P3,  on  which  interviewers  were 
instructed  to  ask  for  an  estimate  of  the  accuracy  of  dollar  figures  if  records 
were  not  consulted.  This  request  was  made  by  showing  the  respondent  a 
so-called  "accuracy  card,"  a  copy  of  which  is  shown  in  the  following 
diagram. 


a. 

Under  $500 

b. 

$500-$999 

c. 

$l,000-$4,999 

d. 

$5,000-$9,999 

e. 

$10,000-$14,999 

f. 

$15,000-$24,999 

g- 

$25,000-$49,999 

h. 

$50,000-$74,999 

i. 

$75,000-$99,999 

]• 

$100,000  and  over 
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Interviewers  were  instructed  to  use  this  card  as  follows:  "Hand  the  card 
to  the  respondent  and  ask  him  to  indicate  which  letter  best  describes  how 
close  the  figure  he  gave  is  to  the  actual  amount.  Explain  that  we  do  not 
expect  people  to  give  exact  figures  from  memory  and  that  it  is  only  natural 
to  have  some  error  when  figures  are  given  in  this  manner."152 

In  terms  of  cooperativeness,  this  accuracy  card  was  not  difficult  to 
administer,  and  few  respondent  refusals  to  use  it  were  reported.  For  ex- 
ample, on  Wave  2  of  Study  P2  when  the  accuracy  card  was  first  used, 
only  one  of  122  respondents  asked  to  use  this  card  in  evaluating  the  ac- 
curacy of  savings  account  balances  refused  to  answer  the  question.  The 
corresponding  refusal  rate  for  the  face  value  of  life  insurance  on  that  wave 
was  2  percent.  On  Wave  1  of  Study  P3,  the  refusal  rate  was  zero  for  both 
savings  accounts  and  life  insurance.  Considering  the  fact  that  the  base 
for  these  percentages  excludes  the  most  cooperative  sample  members  — 

152  Interviewer  Guide  for  Farm  Savings  Panel,  p.  41. 


Table  81.    Cross-Tabulation  of  Selected  Interviewer  Ratings, 
by  Accuracy  and  Completeness  of  Report  of  Validated  Holding 


Study 


Percentage  of 

respondents 

reporting 

holdings 


Percentage  of 

dollar  figures 

accurate  within 

25  percent 


Life  insurance 


PI 

Accuracy 

Excellent 
Good 
Fair 
Poor  or 
very  poor 

70 
65 
58 

42 

49 
66 
50 
50 

Completeness 

Excellent 
Good 
Fair 
Poor  or 
very  poor 

70 
64 
48 
50 

49 
61 
54 
62 

P2 

Accuracy  on 
life  insurance 

Fully 
Fairly 
Not  much, 
very  little 

55 
60 
25 

88 
62 
33 

Completeness  on 
life  insurance 

Fully 
Fairly 
Not  much, 
very  little 

55 
60 
25 

83 
62 
67 

P3 

Willingness  to  give 
financial  information 

Excellent 
Good 
Fair 
Poor  or 
very  poor 

71 
52 
60 
50 

69 
90 
78 
50 
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Table  81.    (concluded) 


Study 


Percentage  of 

respondents 

reporting 

holdings 


Percentage  of 

dollar  figures 

accurate  within 

25  percent 


Savings  accounts 


P2 

Accuracy  on 

savings  accounts 

Fully 
Fairly 
Not  much, 
very  little 

78 
68 
40 

80 
58 

17 

Completeness  on 
savings  accounts 

Fully 
Fairly 
Not  much, 
very  little 

76 
65 
39 

85 

37 
17 

P3 

Accuracy  on 

savings  accounts 

Fully 
Fairly 
Not  much, 
very  little 

34 
23 
24 

78 

33 

0 

Completeness  on 
savings  accounts 

Fully 
Fairly 
Not  much, 
very  little 

30 
24 
33 

69 
0 
0 

Farm  debt 


P3. 


Accuracy  on  debt 


Completeness  on  debt 


Fully 
Fairly 
Not  much, 
very  little 

Fully 

Fairly 
Not  much, 
very  little 


83 
78 

56 

94 

58 
80 


those  giving  dollar  balances  from  records — the  feasibility  of  this  accuracy- 
card  approach  would  seem  to  be  confirmed.153 

Given  the  feasibility  of  this  approach,  the  basic  question  relates  to  its 
value.  A  partial  answer  to  this  question  is  provided  in  Table  82  in  which 
the  answers  obtained  from  this  accuracy  card  are  cross-tabulated  by  ac- 
curacy of  the  data  for  savings  accounts  and  life  insurance  for  Studies  P2 
and  P3  and  for  farm  debt  for  Study  P3. 

As  is  evident  from  this  table,  the  accuracy-card  replies  do  provide 
information  on  the  true  accuracy  of  the  data,  although  a  noticeable  tend- 
ency exists  for  the  accuracy  to  be  understated.  Nevertheless,  figures  rated 
either  exact  or  with  a  low  margin  of  error  are  much  more  likely  to  possess 
high  accuracy  than  those  assigned  larger  margins  of  error.    Figures  re- 

153  This  qualification  need  not  be  applied  in  the  case  of  land  and  buildings, 
for  which  few  farmers  were  in  a  position  to  consult  records. 
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ported  with  large  margins  of  error  are  more  subject  to  large  error  than 
figures  reported  with  small  margins  of  error,  although  instances  are  ap- 
parent of  large  error  ranges  being  assigned  to  figures  which  are  actually 
fully  correct.154  Clearly,  if  the  tendency  to  overstate  could  be  reduced, 
perhaps  through  a  different  form  of  question  wording,  the  value  of  the 
accuracy  card  would  be  enhanced  further.  Even  so,  the  relationships  indi- 
cated in  Table  82  would  suggest  further  experimentation  with  this 
technique. 

Personality  Tests 

Personality  variables  have  been  shown  in  previous  chapters  to  be  re- 
lated to  the  accuracy  with  which  data  are  reported.  In  addition,  the 
analysis  of  the  accuracy  of  the  data  by  interviewer  characteristics  brings 
out  certain  personality  traits  which  may  be  highly  relevant  to  the  relia- 
bility of  the  data.155  Accordingly,  an  attempt  was  made  to  investigate  the 
extent  to  which  personality  of  the  respondent  might  be  related  to  data 
reliability.  For  this  purpose,  the  Edwards  Personal  Preference  Schedule 
was  administered  to  respondents  on  Wave  5  of  Study  P2,  the  same  test 
having  been  administered  somewhat  earlier  to  the  interviewers  in  that 
study.  Using  these  data,  comparisons  are  shown  in  Table  83  between  the 
accuracy  and  completeness  of  the  validated  savings  accounts  and  the  dif- 
ferences between  the  personality  scores  of  the  corresponding  interviewers. 

To  judge  by  these  data,  a  number  of  differences  exist  in  the  ability  of 
interviewers  with  different  personalities  to  obtain  complete  and  accurate 
data  on  savings  accounts.  Most  understandable  is  the  greater  tendency 
for  the  validated  account  to  be  reported,  and  for  more  accurate  informa- 
tion to  be  obtained,  when  the  interviewer  is  more  dominant  than  the 
respondent.  Order  is  another  personality  characteristic  that  appears  to  be 
more  closely  related  to  success  in  obtaining  such  information  than  inter- 
viewer skill.  In  this  case,  the  more  orderly  is  the  respondent  relative  to  the 
interviewer,  the  more  likely  is  the  validated  account  to  be  reported  and  to 
be  reported  accurately. 

Interestingly  enough,  the  data  tend  to  bear  out  the  danger  of  inter- 
viewers becoming  too  friendly  with  respondents,  since  less  accurate  and 
less  complete  information  is  obtained  when  the  respondents  have  higher 
"succorance"  scores,  higher  "nurturance"  scores,  or  higher  "affiliation" 
scores,  any  of  which  may  indicate  a  tendency  toward  a  "gabby"  interview. 

154  Such  instances  are  most  frequent  in  the  case  of  the  face  value  of  life  insur- 
ance, where  the  amounts  involved  tend  to  be  far  larger  than  the  error  ranges  used, 
so  that  the  relative  errors  are  still  small.  Moreover,  as  was  noted  in  Chapter  VI, 
the  term  "face  value  of  life  insurance"  tends  to  be  easily  misunderstood. 

153  Steinkamp  and  Hauck,  ibid.,  Chapter  III. 
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Table  83.   Comparison  of  Average  of  Cumulated  Respondent  Less 

Interviewer  Scores  on  EPPS  and  Accuracy  and  Completeness 

of  Validated  Accounts,  Wave  5,  Study  P2 


Characteristic 

Validated  account 
reported 

Accuracy  of  reported 

account 

(of  SI 00  or  more) 

Yes 

No 

Within 
10  percent 

10  percent 
or  more 

Achievement 

-1.1 

.3 

3.1 

-.8 

.6 

.2 

-2.8 

3.2 

-2.5 

3.3 

1.2 

-3.3 

1.2 

-3.2 

.4 

-.1 

-.4 

1.8 

-.4 

-1.0 

1.9 

-2.1 

4.5 

0 

2.6 

.7 

-3.3 

.3 

-4.1 

.8 

-.9 

.5 

4.5 

1.1 

.3 

-.4 

-2.1 

2.8 

-3.2 

3.7 

.8 

-3.2 

2.2 

-4.8 

.8 

-1.7 

Deference 

Order 

Exhibition 

Autonomy 

Affiliation 

.3 

.6 

.2 

1.1 

1.3 

Intraception 

Succorance 

Dominance 

Abasement 

Nurturance 

Change 

Endurance 

Heterosexuality 

-4.2 
4.3 

-1.4 
2.0 
1.5 

-3.1 
-.1 
-.4 

Aggression 

-.9 

Such  characteristics  as  aggression,  achievement,  and  autonomy  do  not 
appear  to  have  much  relation  to  the  reliability  of  the  data.156 

Asking  Respondents  for  Validation  Permission 

By  far  the  most  useful  means  of  evaluating  the  reliability  of  survey 
data  is  to  obtain  the  corresponding  true  figures  from  external  sources. 
Doing  so  by  selecting  the  sample  from  the  records  of  an  external  source  is 
conceptually  the  best  approach.  Another  way  of  doing  so,  however,  is  to 
attempt  to  obtain  this  information  through  the  cooperation  of  the  respon- 
dent by  asking  him  for  written  permission  to  contact  the  institutions  from 
which  corresponding  reports  of  his  asset  and  debt  holdings  can  be 
obtained. 

This  approach  was  tried  on  Wave  5  of  Study  PI  and  on  Wave  3  of 
Study  P2,  in  both  cases  with  regard  to  savings  account  balances.  In  each 
case,  respondents  were  asked  to  sign  a  form  requesting  their  savings  insti- 
tutions to  provide  by  mail  to  the  project  headquarters  the  balance  in  their 
accounts  as  of  the  date  of  the  interview.  The  basis  for  the  request  was  the 
assertion  that  considerable  uncertainty  existed  regarding  the  extent  to 
which  passbook  balances  coincided  with  records  of  the  savings  institution 

156  Most  puzzling  of  all  is  the  suggestion  in  Table  83  that  accounts  are  re- 
ported with  much  greater  accuracy  when  the  interviewer  achieves  a  higher  score  on 
"heterosexuality"  than  the  respondent! 
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because  of  the  possibility  of  unrecorded  deposits,  accumulated  interest, 
transactions  on  the  date  of  the  interview,  or  balances  representing  hold- 
ings of  more  than  just  the  passbook  holder  (s) . 

The  results  were  mixed.  On  Study  PI,  62  percent  of  131  panel  mem- 
bers approached  with  this  request  signed  the  form,  whereas  on  Study  P2 
somewhat  fewer  panel  members  cooperated  (47  percent) .  Those  who 
refused  to  sign  the  requested  form  did  include  more  nonreporters,  but 
refusal  to  sign  it  did  not  necessarily  imply  refusal  to  provide  financial  data. 
This  is  brought  out  in  the  following  tabulation  relating  to  Study  P2. 

Asked  to  sign  requested  form 
Signed  Did  not  sign 

Reported  account 75%  60% 

Did  not  report  account 25  40 

This  approach  may  also  provide  useful  information  on  an  individual 
basis.  Comparison  of  dollar  balances  of  the  respondents  with  correspond- 
ing balances  obtained  from  the  institutions  revealed  a  fair  number  of  dis- 
crepancies (Table  84).  Since  the  respondent  was  fully  aware  of  the 
validation  procedure,  a  request  for  clarification  could  be  (and  was)  made 
to  the  respondent  or  to  the  institution,  so  that  the  reliability  of  these 
reports  could  be  improved  substantially.  The  fact  remains,  however,  that 
as  with  the  accuracy-card  approach,  this  procedure  provides  only  a  partial 
solution  to  the  problem,  since  by  definition  it  is  not  applicable  if  no  hold- 
ings are  reported. 

Primary  and  Secondary  Validation 

From  an  analytical  point  of  view,  evaluation  of  respondent  reports  — 
whether  by  selecting  the  sample  from  the  institutions  or  by  secondary 
validation  —  is  an  ideal  approach  to  data  reliability.  Since  its  feasibility 
has  already  been  considered  extensively,  there  is  no  need  for  further  dis- 
cussion. It  should  be  stressed,  however,  that  secondary  validation  possesses 
the  disadvantage  of  being  unable  to  detect  complete  nonreporting.  It 
would  be  most  useful  if  names  of  sample  members  could  be  circulated  to 
all  institutions  of  a  given  type  in  a  particular  area  (as  was  apparently  done 
in  another  financial  study) .  This  procedure  would  still  permit  population 
probability  sampling,  and  at  the  same  time  it  would  greatly  minimize  the 
danger  of  overlooking  nonreported  holdings. 

Reducing  Residual  Nonsampling  Variance 

Under  the  circumstances,  if  the  overall  efficiency  of  sample  surveys  is 
to  be  increased,  the  reduction  of  nonsampling  errors  merits  primary  con- 
sideration.   To  a  certain  extent,  such  reduction  may  be  accomplished 
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through  the  improvement  of  survey  procedures.  Yet,  such  improvements 
can  only  go  so  far,  and  if  the  findings  presented  in  this  volume  are  valid, 
a  substantial  residual  nonsampling  variance  is  likely  to  remain  in  most 
survey  operations.  Essentially,  the  reason  for  this  derives  from  the  fact 
that  the  inducement  to  cooperate  on  any  survey  depends  not  only  on  the 
subject  of  the  survey  and  the  manner  in  which  it  is  being  conducted  but 
also  on  the  personality  and  background  of  the  sample  member  as  well  as 
on  the  personal  and  physical  circumstances  in  which  the  interview  is 
attempted.  These  latter  sets  of  variables  may  give  rise  to  either  response 
or  nonresponse  errors,  as  has  been  illustrated  in  previous  chapters. 

Moreover,  response  and  nonresponse  errors  subject  to  survey  control  in 
theory  may  not  always  be  so  in  practice.  Thus,  skepticism  of  the  value  of 
a  study  or  doubts  about  the  confidentiality  of  the  data  can  presumably  be 
resolved  in  each  particular  case  by  a  proper  survey  approach.  However, 
the  nature  of  this  "proper  survey  approach"  will  vary  from  one  sample 
member  to  another,  so  the  interviewer  is  hardly  likely  to  select  always 
"the"  approach  for  dealing  with  each  case.  For  example,  if  an  inter- 
viewer attempts  to  induce  an  advocate  of  laissez-faire  to  cooperate  because 
the  data  will  be  useful  for  government  economic  planning,  he  may  be 
rebuffed,  even  though  the  same  approach  will  be  effective  on  other  people. 

As  a  result,  nonsampling  field  errors  may  be  said  to  be  composed  in 
practice  of  errors  that  could  be  and  were  avoided  by  proper  survey  meth- 
ods, of  errors  related  to  the  survey  methods  but  which  were  not  avoided, 
and  of  errors  outside  of  survey  control.  Clearly,  improved  survey  pro- 
cedures help  to  limit  the  second  type  of  error.  At  the  same  time,  it  seems 
equally  clear  that  errors  of  the  second  type  will  occur  and  that  errors  of 
the  third  type  can  not  be  handled  by  the  usual  survey  procedures.  For 
such  errors,  something  new  must  be  added. 

The  nature  of  this  new  ingredient  becomes  evident  when  we  consider 
that  in  the  main  such  errors  are  not  random  but  occur  in  certain  system- 
atic ways.  These  systematic  ways  reflect  the  factors  mentioned  previ- 
ously that  cause  such  errors  —  personality  of  the  sample  member,  back- 
ground, circumstances  of  the  interview,  and  so  forth.  On  the  other  hand, 
in  practice  it  is  hardly  feasible  to  distinguish  between  the  different  types 
of  errors,  in  part  because  of  lack  of  data  and  in  part  because  errors  pre- 
sumably within  survey  control  also  tend  not  to  be  random.  These  errors 
may  also  be  related  to  a  respondent's  characteristics,  such  as  respondent 
misunderstanding  arising  from  interaction  between  the  use  of  technical 
terms  and  low  educational  background,  as  well  as  to  survey  conditions. 

Furthermore,  since  these  errors  are  also  systematic,  to  the  extent  that 
they  exist  there  is  a  need  for  their  detection  and  correction  as  much  as 
there  is  for  errors  outside  of  survey  control.    In  other  words,  the  need  is 
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for  a  means  of  identifying  response  and  nonresponse  errors  as  they  occur 
and  relating  such  errors  to  relevant  characteristics  of  the  sample  member, 
of  the  interviewer,  and  of  the  interview  situation.  If  such  relationships 
can  be  derived  and  if  they  are  reasonably  stable,  a  basis  then  exists  for 
adjusting  survey  data  for  the  effects  of  such  errors. 

The  derivation  of  such  relationships  within  a  general  analytical  frame- 
work points  toward  a  second  major  direction  which  future  research  on 
nonsampling  errors  needs  to  follow.  For  if  this  variance  cannot  be  re- 
duced through  better  survey  procedures,  other  procedures  must  be  intro- 
duced to  cope  with  the  problem,  and  the  most  promising  of  such  other 
procedures  would  seem  to  be  the  combination  of  external  information  with 
survey  data  through  the  use  of  multivariate  analysis.  Different  ways  by 
which  this  combination  may  be  achieved  are  being  explored.  This  is  not 
the  place  to  describe  these  methods  in  detail;  indeed,  much  more  work  is 
needed  before  this  can  be  done.  It  would  seem  appropriate  here,  how- 
ever, to  outline  the  methods  that  are  being  considered. 

Alternative  Approaches 

Two  alternative  approaches  are  being  investigated  for  transferring  in- 
formation on  nonsampling  errors  from  a  validation  sample  to  a  population 
probability  sample.  Both  of  these  approaches  are  predicated  on  the  selec- 
tion of  two  largely  replicating  samples  from  the  population  being  studied : 
Sample  1  (St) ,  representing  a  probability  sample  of  consumer  units  from 
the  general  population,  and  Sample  2  (S2) ,  a  probability  selection  of  con- 
sumer patrons  of  cooperating  financial  institutions,  the  latter  preferably 
also  selected  by  probability  methods.  Identical  questionnaires  and  inter- 
viewing methods  are  employed  on  both  samples.  As  a  result,  the  valida- 
tion data  applied  to  S2  yield  the  errors  in  reports  of  the  members  of  S2, 
as  well  as  information  not  obtained  from  the  refusals  and  the  nonrespon- 
dents  in  £2.  The  problem  then  becomes  one  of  making  use  of  this  infor- 
mation to  correct  the  reports  of  the  members  of  £1  for  both  response  and 
nonresponse  errors. 

One  approach  is  what  may  be  called  the  "error-determinant"  model, 
or  Model  A.  It  is  based  on  determining,  with  the  aid  of  the  S2  data,  the 
factors  that  account  for  response  errors  and  nonresponse  bias  and  making 
use  of  these  results  to  detect  and  correct  corresponding  errors  in  Si.  Since 
previous  chapters  have  shown  the  most  important  form  of  response  error 
to  be  nonreporting  of  holdings,  the  effectiveness  of  this  model  will  rest 
primarily  on  its  ability  to  pinpoint  nonreporters  in  Sx.  Once  nonreporters 
in  S-l  are  identified,  nonowners  of  the  given  asset  in  Sx  can  be  obtained  as 
a  residual  —  as  those  sample  members  who  reported  not  owning  that  asset 
and  who  were  not  identified  as  nonreporters. 
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The  second  approach,  Model  B,  is  an  "owner-determinant"  model. 
This  model  seeks  to  cope  directly  with  the  basic  structural  characteristic 
that  differentiates  Sx  from  S2,  namely,  the  fact  that  by  definition  all  the 
members  of  S2  own  the  particular  asset,  whereas  this  is  not  true  of  all  of 
the  members  of  S^  Consequently,  the  rationale  underlying  Model  B  is  to 
apply  analytical  methods  to  the  validation  data,  in  conjunction  with  data 
for  the  general  population,  to  obtain  a  means  of  discriminating  between 
owners  and  nonowners  in  St.  With  the  aid  of  such  methods,  the  members 
of  iSx  are  classified  as  owners  or  nonowners  of  the  particular  asset  (or, 
alternatively,  as  each  being  an  owner  with  a  given  probability).  Non- 
reporters  are  then  obtained  as  a  residual  —  as  those  members  of  £1  classi- 
fied as  owners  by  the  analytical  technique  but  who  deny  holding  that  asset. 

Each  model  has  advantages  and  disadvantages.  The  error-determinant 
model  requires  the  derivation  of  a  set  of  functions  relating  response  errors 
to  the  characteristics  of  the  respondent,  interviewer,  and  interview  situa- 
tion. It  also  requires  a  means  of  distinguishing  nonreporters  in  St  from 
true  nonowners.  On  the  other  hand,  the  technique  possesses  the  advan- 
tage of  not  requiring  any  data  other  than  those  collected  in  the  interviews. 

The  owner-determinant  model  requires  a  technique  for  discriminating 
between  owners  and  nonowners  of  the  particular  asset,  a  basic  structural 
characteristic  of  the  population.  (It  might  be  argued  that  if  such  a  rela- 
tionship could  be  found,  much  of  the  basic  problem  would  no  longer 
exist.)  For  obtaining  such  a  relationship,  the  model  requires  external 
population  data  comparable  with  the  survey  data.  Moreover,  it  requires 
an  independent  estimate  of  the  aggregate  number  of  owners  of  that  asset 
in  the  population. 

Estimation  Procedures 

These  two  models  are  being  tested  in  various  ways.  Two  variations  of 
the  error-determinant  model  have  been  formed.  One  variation  consists  of 
a  set  of  five  equations  which  attempts  to  identify  and  correct  response 
errors  classified  into  three  categories:  nonreporting,  reporting  of  owner- 
ship but  refusal  of  amount,  and  reporting  of  amount  but  with  unknown 
error.  A  second  form  compresses  this  model  into  two  equations,  with  one 
equation  seeking  to  pinpoint  those  sample  members  making  errors  of  more 
than  a  specified  proportion  and  a  second  equation  correcting  for  this  error, 
given  its  existence. 

Both  of  these  variations  of  the  error-determinant  model  are  being 
tested  in  three  ways.  One  way  is  by  seeking  standard  discriminant  or  mul- 
tiple regression  relationships,  classifying  sample  members  as  falling  into 
one  category  or  the  other  and  then  assigning  amounts  with  the  aid  of 
other  functions,  as  required.    The  second  approach  is  to  use  the  same 
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estimation  methods  as  with  the  first  approach,  but  in  the  classification  of 
reporters-nonreporters  and  of  errors-nonerrors  to  assign  each  sample  mem- 
ber a  probability  of  being  a  nonreporter  (or  reporting  with  error) ,  and  to 
use  these  probabilities  as  a  basis  for  distributing  amounts  in  the  various 
size-of-holding  classes.  The  third  approach  is  to  use  probabilities  rather 
than  regression  methods  for  pinpointing  nonreporters  and  respondents 
reporting  with  error.  This  approach  requires  the  construction  of  a  tree 
diagram  instead  of  a  regression  function,  which  has  the  advantage  of 
maximum  flexibility  in  forming  categories  in  introducing  interactions  but 
which  uses  up  data  very  quickly. 

The  owner-determinant  model  is  being  tested  by  the  use  of  tree  dia- 
grams, since  data  are  not  available  for  deriving  regression  functions  to 
segregate  owners  from  nonowners. 

The  error-determinant  model:  Variant  A.  This  variant  is  the  five- 
equation  model  which  attempts  to  detect  response  errors  classified  into 
three  categories.  The  model  can  be  summarized  briefly,  using  the  follow- 
ing notation : 

D  is  a  discrepancy  of  any  kind  between  the  true  figure  and  reported  figure, 
E  is  an  error  in  the  amount  reported  by  the  respondent, 
H  is  a  true  holder  of  the  particular  asset, 
R'  is  a  nonreporter  (R,  a  reporter) , 
W  is  a  respondent  acknowledging  ownership  of  an  asset  but  refusing  the 

amount, 
Z  is  a  set  of  explanatory  variables  relating  to  the  characteristics  of  the  re- 
spondent, interviewer,  and  interview  situation, 
/  is  a  likelihood  of  a  particular  amount,  and 
s  is  a  dollar  amount. 

With  these  definitions,  the  five-equation  model  is  as  follows: 
Correction  for  nonreporters : 

(1)  l(R')  =/1(Zi),and 

(2)  s(R')  =/,(Z4). 

Amount  withheld  of  acknowledged  holding: 

(3)  s(W)  =f3(Zi). 

Correction  for  errors  in  reported  amounts : 

(4)  1(E)  =/4(Zi),and 

(5)  s(E)  =/5(Zi). 

The  parameters  of  these  equations  are  to  be  estimated  from  the  val- 
idation data,  so  that  in  all  instances  the  equations  contain  conditional 
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dependent  variables,  namely,  variables  given  that  the  respondent  does  own 
that  asset.  Therefore,  when  these  equations  are  applied  to  an  Sx  sample, 
Function  (1)  has  to  be  adjusted  by  a  separately  determined  probability 
that  the  nonreporter  in  Sj_  really  holds  some  of  that  asset  and  is  not  telling 
the  truth  when  he  reports  zero  holdings.  In  effect,  for  S1}  Function  ( 1 ) 
becomes 

(1')  l(R')  =  1(H)  •  l(R'\H), 

which  assumes  independence  between  the  two  relations. 

Two  approaches  to  estimating  1(H)  are  suggested.  One  approach  is  to 
seek  an  independent  function,  in  the  form  of  a  tree  diagram,  expressing  the 
relative  frequency  of  ownership  of  the  particular  asset  in  the  population 
for  different  combinations  of  relevant  demographic  and  other  variables. 

The  second  approach  rests  on  obtaining  an  independent  estimate  of 
the  number  of  nonreported  accounts  in  Sx,  using  S2  data.  If  p  is  the  pro- 
portion of  all  family  units  in  S2  not  reporting  one  or  more  validated 
accounts,  the  proportion  of  respondents  in  £1  not  reporting  validated  or 
non validated  accounts  may  be  estimated  as  p(\  +  k) ,  where  A;  is  a  frac- 
tion between  zero  and  one  reflecting  the  extent  to  which  families  other 
than  those  not  reporting  validated  accounts  may  not  have  reported  non- 
validated  accounts.  (Cross-validation  data  from  prior  panel  operations 
should  be  of  value  in  specifying  reasonable  limits  for  k.)  The  number  of 
nonreporters  in  Sx  as  a  proportion  of  reporters  is,  then : 

\-p{\+k)- 

Function  ( 1 )  would  be  applied  to  S{,  yielding  estimated  values  of  the 
dependent  variable  for  each  respondent.  These  estimates  would  be 
arrayed  in  descending  order  and  the  top  Nx  respondents  designated  as 
nonreporters,  where  N  is  the  number  of  sample  members  in  Sx. 

Variant  B.  This  approach  seeks  to  simplify  the  five-equation  model 
into  three  equations.  One  equation  represents  the  likelihood  of  any  dis- 
crepancy whatsoever  (including  nonreporting) ,  a  second  equation  esti- 
mates the  true  amount  (given  that  a  discrepancy  exists) ,  and  a  third  equa- 
tion estimates  amounts  refused  —  this  is  Function  (3)  once  more.  The 
first  two  equations  can  be  represented  as  follows: 

(6)  1(D)  =/.(Z«),and 

(7)  s(D)  =fr(Zi). 

The  problem  of  correction  for  true  nonownership  in  Si  is  the  same  as 
before,  and  could  presumably  be  handled  by  the  same  methods. 

Estimation  of  function  parameters.  Standard  multiple  regression  meth- 
ods would  be  used  in  all  instances  to  estimate  the  parameters  of  the  func- 
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tions  containing  a  continuous  dependent  variable- — -Functions  (2),  (3), 
(5) ,  and  (7) .  The  alternative  procedures  enter  into  the  estimation  of  the 
likelihood  functions.  These  alternatives  are  as  follows: 

The  first  is  to  estimate  the  parameters  of  these  functions  by  the  least- 
squares  method,  establish  a  cut-off  point  as 

.         l(R)+l(R') 
lo  —  2  ' 

and  classify  respondent  i  as  a  nonreporter  if  U  is  less  than  l0  and  as  a 
reporter  if  U  exceeds  l0.  For  the  nonreporters  so  classified,  use  Function 
(2)  to  obtain  an  estimate  of  the  amount  not  reported,  which  is  then  as- 
signed to  the  sample  member. 

The  second  alternative  is  to  estimate  the  parameters  of  these  functions 
by  the  least-squares  method.  Assign  to  each  observation  a  probability  of 
being  a  nonreporter  based  on  the  following  relation : 

('pit)  v{l)  dl 

p(R'\l1<l<k)=-±-j2 , 

/     v{l)  dl 

where  p(l)  is  the  probability  of  a  respondent  with  the  value  /  saying  that 
he  does  not  hold  that  asset  (given  that  he  does  hold  it)  and  v(l)  is  the 
proportion  of  people  with  estimated  values  between  the  two  limits,  lx  and 
l2.  After  probabilities  of  nonreporting  are  assigned,  an  estimated  amount 
not  reported  is  obtained  from  Function  (2)  for  each  respondent,  as  well 
as  a  standard  deviation  of  this  estimate.  The  resulting  amount  is  allocated 
in  a  size-of-holding  distribution  using  these  probabilities  on  the  basis  of 
a  lognormal  distribution;  this  distribution  would  seem  to  approximate  the 
true  distribution  of  holdings  better  than  a  normal  distribution.  (Howard 
Raiffa  has  also  suggested  a  means  of  using  these  data  for  regression  anal- 
ysis, although  the  exact  details  of  the  method  have  yet  to  be  worked  out.) 

The  third  approach  calls  for  the  construction  of  tree  diagrams  rather 
than  estimating  the  parameters  of  the  functions  by  regression  methods. 
Briefly,  the  approach  involves  selecting  the  most  relevant  variables  felt 
to  influence  nonreporting,  setting  up  meaningful  categories  of  each  vari- 
able, and  combining  the  categories  for  different  variables  to  obtain  the 
maximum  discrimination  between  cells  for  the  relative  frequency  of  non- 
reporters. 

The  owner-determinant  model.  This  model  seeks  to  discriminate 
first  between  owners  and  nonowners  in  S^  and  then  obtain  nonreporters 
as  a  residual.  Because  of  the  data  problems  associated  with  the  model,  its 
specification  is  best  outlined  by  a  series  of  steps,  as  follows: 
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( 1 )  Estimate  the  total  number  of  savings  units  owning  the  particular 
asset  in  the  population. 

(2)  Using  the  £2  data,  obtain  distributions  of  the  number  of  owners 
of  that  asset  in  the  population  for  whatever  combinations  of  characteristics 
are  deemed  most  relevant  for  segregating  owners  from  nonowners. 

(3)  By  relating  these  distributions  to  comparable  census  or  other  ex- 
ternal data,  obtain  relative  frequencies  of  ownership  of  that  asset  by  popu- 
lation characteristics,  in  the  form  of  a  tree  diagram. 

(4)  Use  the  resulting  tree  diagram  to  segregate  owners  from  non- 
owners  in  Sr.  As  with  the  error-determinant  model,  this  classification 
could  be  made  on  a  one-zero  basis  or  by  assigning  each  respondent  a 
designated  probability  of  ownership. 

(5)  Identify  nonreporters  in  Sx  as  those  who  have  reported  no  hold- 
ings and  are  labeled  as  owners  on  the  basis  of  the  tree  diagram. 

(6)  Estimate  the  amount  not  reported  by  using  Function  (2) . 

Undoubtedly  other  forms  of  model  specification  and  estimation  pro- 
cedures will  be  developed  with  further  work.  The  forms  outlined  here  are 
not  meant  to  be  definitive  but  are  intended  instead  to  indicate  the  types 
of  models  that  are  being  considered. 

Summary  Comments 

The  observations  and  experiments  reported  in  this  chapter  provide 
evidence  that  improvements  which  could  lead  to  considerable  increases  in 
data  reliability  are  possible  in  current  survey  procedures.  Judging  by  the 
state  of  the  art  today,  the  largest  potentials  for  such  improvement  would 
seem  to  lie  in  interviewer  selection  and  supervision  and  in  devising  means 
of  evaluating  the  reliability  with  which  data  are  collected.  Little  has  been 
accomplished  in  the  latter  connection;  in  the  former  respect,  much  re- 
mains to  be  done  despite  the  widespread  attention  interviewer  selection 
and  supervision  have  received  in  the  past.  In  both  these  and  other  areas, 
the  experiments  described  in  this  chapter  ■ — ■  although  exploratory  in 
nature  —  indicate  directions  in  which  future  gains  may  be  substantial, 
gains  which  would  be  applicable  far  beyond  the  scope  of  consumer  finan- 
cial surveys. 

Perhaps  the  largest  source  of  improvement,  however,  lies  in  the  inte- 
gration of  external  data  with  the  survey  data.  Some  methods  for  doing 
this  have  been  outlined,  but  work  on  such  procedures  is  as  yet  in  its  initial 
stages. 

In  closing  this  chapter,  the  exploratory  nature  of  the  experiments  car- 
ried out  should  be  stressed.  Their  primary  objective  was  not  to  obtain 
definitive  results  but  to  investigate  the  feasibility  of  different  means  of 
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improving  survey  procedures.  Because  of  the  limited  scope  of  the  oper- 
ations, many  of  the  ideas  had  to  be  tested  on  an  ad  hoc  basis.  Even  when 
controlled  experiments  were  conducted,  their  restricted  scale  led  to  results 
that  are  primarily  suggestive.  The  full  value  of  these  alternative  methods 
can  only  be  established  by  large-scale  tests. 


VIII.  SUMMARY  AND  CONCLUSIONS 


This  final  chapter  is  divided  into  three  parts.  The  first  part  presents  a 
general  overview  of  the  principal  findings  in  this  volume.  The  significance 
of  these  findings  to  surveys  other  than  those  relating  to  consumer  finances 
is  discussed  in  the  second  part  of  the  chapter.  Suggested  future  approaches 
to  the  subject  of  nonsampling  errors  are  discussed  in  the  final  part,  which 
also  presents  recommendations  for  further  research. 

Summary  of  Results 

At  the  outset,  ways  in  which  nonsampling  errors  can  arise  in  consumer 
financial  surveys  were  outlined,  and  a  framework  was  provided  of  the 
effects  that  such  errors  might  have  on  survey  data.  Among  other  things,  it 
was  stressed  that  in  theory  nonsampling  errors  may  affect  the  reliability  of 
survey  data  as  well  as  the  accuracy  of  such  data,  and  the  former  effect  was 
shown  to  exist  even  when  sample  estimates  of  population  parameters  were 
unbiased.  It  was  also  noted  that  there  need  be  no  correlation  between  the 
two  types  of  errors. 

The  empirical  studies  presented  in  the  later  chapters  indicate  in 
striking  fashion  that  nonsampling  errors  are  not  simply  a  matter  of  theory, 
but  do  in  fact  exist  and  are  mainly  responsible  for  the  pronounced  tend- 
ency of  survey  data  to  understate  aggregates,  as  was  noted  in  Chapter  II. 
Not  only  was  this  bias  present  in  the  survey  data,  but  in  many  instances 
the  contribution  of  nonsampling  errors  to  the  total  variance  in  the  data 
was  so  large  as  to  render  meaningless  confidence  intervals  computed  by 
the  usual  statistical  formulas.  As  was  brought  out  in  Chapters  III  and  IV, 
the  magnitude  of  this  type  of  error  tends,  if  anything,  to  increase  with 
sample  size,  because  such  increases  tend  to  reduce  the  sampling  variance 
while  having  no  effect  on  the  nonsampling  variance. 

The  magnitude  and  relative  importance  of  these  nonsampling  errors 
were  seen  to  vary  substantially  with  the  type  of  financial  holding  and  with 
the  type  of  data  sought  about  that  holding.  In  general,  holdings  con- 
sidered more  confidential  than  others  tended  to  be  subject  to  larger  errors 
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than  other  holdings.  Much  larger  biases  and  larger  relative  nonsampling 
variances  were  obtained  for  savings  accounts  and  personal  loans  than  for 
checking  accounts  and  automobile  debt.  With  the  latter  type  of  holdings, 
the  principal  effect  of  nonsampling  errors  appeared  to  be  on  the  reliability 
of  the  estimates  rather  than  on  the  accuracy  of  the  estimates.  Particularly 
interesting  is  the  finding  that  roughly  three-fourths  of  the  variance  due  to 
discrepancies  in  checking  account  balances  was  due  to  nonsampling  errors 
rather  than  to  the  frequently  cited  problem  of  check-float. 

The  analysis  of  the  panel  data  in  Chapter  VI  showed  that  nonsam- 
pling errors  tend  to  persist  throughout  a  survey  operation,  although  their 
relative  influence  declines  somewhat  over  time.  Respondent  rapport  was 
shown  to  improve  over  time.  Hence,  although  appreciable  mortality  was 
experienced  on  these  panel  operations,  the  improved  accuracy  of  those 
who  did  remain  in  the  panel  more  than  offset  the  increased  atypicalness 
of  the  panel.  As  a  result,  aggregate  data  obtained  from  the  panel  became 
more  accurate  over  time,  despite  the  increasing  lack  of  representativeness. 

How  do  nonsampling  errors  arise?  A  simple  segmentation  technique 
employed  in  these  chapters  showed  that  by  far  the  principal  cause  of  these 
errors  was  the  failure  of  the  sample  member  to  acknowledge  ownership  of 
a  particular  asset  or  debt.  Reinforcing  this  phenomenon  was  a  tendency 
for  nonrespondents  and  for  those  who  refused  to  give  dollar  figures  to 
have  larger  holdings  than  other  sample  members.  In  contrast,  when  hold- 
ings were  reported  and  balances  given,  the  figures  were  likely,  if  anything, 
to  be  overstated. 

Various  experiments  seeking  to  cope  with  response  and  nonresponse 
errors  through  the  improvement  of  survey  procedures  were  considered  in 
the  preceding  chapter,  and  the  results  of  a  number  of  tests  designed  to 
explore  the  practicability  of  such  improvements  were  presented.  As  was 
brought  out  in  that  chapter,  the  main  potential  for  improvement  of  survey 
techniques  would  seem  to  lie  in  better  interviewer  control  and  in  seeking 
means  of  measuring  the  reliability  with  which  data  are  reported.  In  the 
latter  connection,  the  validation  technique  was  once  more  seen  to  offer 
considerable  promise.  The  same  methodology  can  serve,  on  the  one  hand, 
as  a  basis  for  minimizing  the  occurrence  of  nonsampling  errors  and,  on  the 
other  hand,  as  a  basis  for  detecting  and  correcting  such  errors  as  they 
occur. 

Significance  for  Other  Types  of  Surveys 

The  applicability  of  the  present  findings  to  other  types  of  surveys  de- 
pends to  a  large  measure  on  the  similarity  of  the  data  sought  in  these 
surveys  to  consumer  financial  data.  Indeed,  the  type  of  survey  is  not  so 
important  as  the  type  of  data  being  collected.  In  effect,  two  types  of  data 
may  be  distinguished  —  factual  data  and  nonfactual  data,  the  latter  cate- 
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gory  comprising  principally  attitudes  and  opinions.  The  applicability  of 
the  present  findings  to  these  two  types  of  survey  data  may  be  considered 
with  regard  to  each  of  the  three  principal  areas  in  which  the  findings  may 
be  classified:  the  use  of  validation  techniques,  the  measurement  of  non- 
sampling  errors,  and  the  improvement  of  survey  procedures. 

Use  of  Validation  Techniques 

For  validation  techniques  to  be  applicable  to  survey  data,  two  condi- 
tions have  to  be  fulfilled.  First,  a  true  answer  must  exist  in  an  opera- 
tional sense.  Second,  a  means  must  be  available  of  ascertaining  this  true 
answer  so  that  the  respondent  report  can  be  compared  with  it. 

These  two  conditions  would  seem  to  be  fulfilled  for  most  types  of  fac- 
tual data,  although  at  times  ingenuity  may  be  required  to  construct  the 
answer  from  external  sources  of  information.  For  many  data  on  economic 
behavior,  such  as  employment  status,  labor-force  participation,  and  prices 
paid  for  goods  or  services,  corroborative  data  can  be  obtained  from  the 
other  party  to  the  transaction,  e.g.,  the  employer  or  the  merchant.  This 
is  also  possible  for  expenditure  data  to  the  extent  that  people  maintain 
charge  accounts  or  their  purchases  are  reported  to  state  agencies  ( as  in  the 
case  of  a  house  or  an  automobile)  or  to  manufacturers  for  warranty  pur- 
poses. For  other  types  of  expenditures,  different  approaches  may  be  re- 
quired, ranging  from  observation  of  shopping  behavior  to  the  inclusion,  in 
a  panel  operation,  of  internal  consistency  checks. 

Much  the  same  situation  holds  with  regard  to  factual  data  of  a  non- 
economic  nature.  Thus,  validation  data  have  been  used  to  check  the 
reliability  of  consumer  responses  on  illnesses  and  hospital  visits.157  Similar 
methods  could  be  applied  to  personal  characteristics  such  as  age  and 
education  and  to  a  wide  variety  of  other  behavioral  phenomena  ranging 
from  magazine  readership  to  job  mobility.  In  general,  validation  methods 
would  seem  feasible  with  any  type  of  data  for  which  some  record  of  a 
transaction  is  kept  by  a  party  other  than  the  consumer.  This  other  party 
will  usually  be  a  business  firm  or  a  governmental  agency,  although  in  some 
situations  the  other  party  may  be  another  consumer. 

The  applicability  of  validation  techniques  to  nonfactual  data  is  much 
more  difficult  but  may  not  be  impossible.  The  primary  problem  in  such 
a  case  is  likely  to  be  the  formulation  of  an  operational  definition  of  a  true 
answer.  The  basis  for  such  a  definition  must  clearly  be  some  sort  of  ob- 
jective information  relating  to  the  expressed  opinion  or  attitude.  Such 
information  may  be  obtainable  if  the  opinion  or  attitude  is  reflected  in 

157  United  States  National  Health  Survey,  Comparison  of  Hospitalization  in 
Three  Survey  Procedures,  Public  Health  Service  Publication  No.  584  (Washington: 
U.  S.  Department  of  Health,  Education  and  Welfare,  1963). 
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past  and  current  behavior.  For  example,  if  a  business  executive  reports 
that  his  firm  expects  higher  sales  in  the  next  three  months,  a  partial  check 
for  this  statement  could  be  based  on  the  firm's  production  plans.  If  a 
consumer  indicates  a  preference  for  custom-made  clothes,  some  idea  of 
the  meaningfulness  of  this  preference  can  be  obtained  by  ascertaining  what 
sort  of  clothes  have  been  purchased  in  the  past. 

In  still  other  instances,  when  no  form  of  factual  check  can  be  made,  an 
approximation  to  a  true  answer  may  sometimes  be  obtained  by  means  of 
consistency  checks.  Thus,  if  a  public  figure  expresses  sympathy  for  the 
labor  movement,  this  sentiment  can  be  compared  with  past  statements 
that  he  might  have  made  on  the  same  subject.  To  be  sure,  consistency 
does  not  assure  validity,  but  in  many  instances  the  two  concepts  are  close 
enough  to  be  considered  interchangeable. 

Although  the  applicability  of  validation  techniques  to  nonfactual  data 
may  not  be  always  possible,  such  techniques  can  be  applied  at  times  even 
to  such  data,  if  sufficient  advance  thought  is  given  to  the  planning  of  the 
survey.  The  more  likely  is  the  subject  under  study  to  bring  about  non- 
sampling  errors,  the  more  thought  should  be  given  to  this  question. 

Measurement  of  Nonsampling  Errors 

The  practicality  of  the  formulas  for  the  sampling  and  nonsampling 
variances  presented  in  Chapter  I  depends  to  a  large  extent  on  the  avail- 
ability of  validation  data.  Yet,  even  when  validation  data  are  not  avail- 
able, estimates  of  nonsampling  variance  can  often  be  obtained  through 
proper  sample  design,  as  has  been  illustrated  in  past  studies  attempting  to 
measure  interviewer  variability.158  Techniques  of  the  latter  type  would 
seem  to  be  virtually  essential  for  some  types  of  nonfactual  data. 

Regardless  of  the  methods  used,  the  findings  in  this  volume  tend  to 
underscore  the  need  for  measuring  these  nonsampling  errors.  These  find- 
ings suggest  that  a  low  level  of  bias  in  an  aggregative  sense  does  not 
necessarily  imply  that  nonsampling  errors  are  of  negligible  importance. 
Rather,  consideration  must  be  given  to  the  possible  presence  of  nonsam- 
pling errors  in  individual  observations  and  to  the  effect  of  these  errors  on 
the  reliability  of  the  sample  data. 

The  present  findings  suggest  that  the  influence  of  these  nonsampling 
errors  tends  to  increase  with  the  sensitivity  of  the  information  being  sought. 

158  For  example,  R.  Franzen  and  R.  J.  Williams,  "A  Method  for  Measuring 
Error  Due  to  Variances  Among  Interviewers,"  Public  Opinion  Quarterly,  Vol.  20, 
No.  3  (Fall,  1956),  pp.  587-92;  W.  F.  F.  Kemsley,  "Interviewer  Variability  and 
a  Budget  Survey,"  Applied  Statistics,  Vol.  9,  No.  2  (June,  1960),  pp.  122-28; 
and  Leslie  Kish,  "Studies  of  Interviewer  Variance  for  Attitudinal  Variables," 
Journal  of  the  American  Statistical  Association,  Vol.  57,  No.  297  (March,  1962), 
pp.  92-115. 
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There  is  little  reason  to  doubt  that  the  same  relationship  will  be  found  to 
exist  for  other  types  of  data  as  well,  including  material  relating  to  opinions 
and  attitudes.  However,  sensitivity  may  not  be  the  only  variable  related 
to  the  incidence  of  nonsampling  errors.  Other  variables  undoubtedly  exist, 
relating  to  the  adequacy  of  interviewer  training,  the  nature  of  the  popula- 
tion being  studied,  the  organization  of  the  questionnaire,  and  other  as- 
pects of  the  survey  operation.  The  influence  of  these  other  variables  may 
well  vary  with  the  type  of  information  being  sought. 

Improvement  of  Survey  Procedures 

The  need  for  improving  survey  procedures  has  been  widely  recognized, 
and  many  of  the  ideas  explored  in  Chapter  VII  were  taken  from  experi- 
ences with  other  types  of  survey  data.  For  this  reason,  most  findings  in 
that  chapter  would  be  applicable  directly  to  other  types  of  surveys.  This  is 
particularly  true  of  the  findings  relating  to  the  data-collection  method, 
interviewer  training  and  compensation,  and  the  measurement  of  data 
reliability.  In  the  latter  respect,  the  use  of  records  and  interviewer  ratings 
of  reliability  of  information  would  seem  to  offer  considerable  promise  for 
improving  the  reliability  of  factual  data  collected  on  future  surveys.  To  a 
more  limited  extent,  interviewer  ratings  may  also  be  of  use  in  evaluating 
the  reliability  of  nonfactual  data,  such  as  by  throwing  light  on  the  degree 
of  thought  given  to  particular  replies. 

In  certain  other  respects,  the  findings  in  Chapter  VII  will  be  of  more 
restricted  applicability.  Thus,  the  apparent  lack  of  influence  of  monetary 
gifts  offered  by  universities  does  not  mean  that  the  same  procedure  may 
not  be  considerably  more  successful  when  data  are  collected  by  a  com- 
mercial organization.159  Asking  for  holdings  or  asking  for  change  may  be 
a  crucial  factor  in  determining  the  reliability  with  which  financial  data 
are  recorded  by  consumers,  but  it  may  be  of  little  consequence  with  regard 
to  such  data  as  prices  or  occupation. 

The  effects  of  a  particular  approach  are  bound  to  vary  substantially 
with  the  nature  of  the  survey,  but  the  possible  sources  of  nonsampling 
error  are  likely  to  be  much  the  same  in  any  survey.  For  this  reason,  any 
improvements  that  can  be  effected  in  one  type  of  survey  are  bound  to 
have  at  least  limited  applicability  to  other  survey  operations  as  well.  At 
the  very  least,  these  results  should  serve  as  the  basis  for  experimentation  in 
other  types  of  surveys. 

159  It  should  be  noted,  however,  that  a  recent  study  of  gifts  and  incentives 
offered  commercially  suggests  that  they  may  have  little  effect  on  the  reliability  of 
the  data,  within  a  wide  range  of  offers.  See,  Seymour  Sudman,  "On  the  Accuracy 
of  Recording  of  Consumer  Panels,"  unpublished  Ph.D.  dissertation,  University  of 
Chicago,  1963. 
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All  things  considered,  it  is  clear  that  improvement  of  survey  procedures 
may  not  be  sufficient  to  cope  with  the  problem  of  nonsampling  errors  in 
many  types  of  surveys;  indeed,  it  may  not  even  be  the  most  effective  way 
of  doing  so.  Perhaps  most  promising  of  all  would  seem  to  be  the  second 
approach  of  seeking  ways  of  integrating  survey  data  with  related  informa- 
tion obtained  from  other  sources.  In  this  respect,  the  validation  data  are 
ideal,  and  some  means  of  using  them  in  conjunction  with  survey  data 
were  outlined  in  Chapter  VII.  The  methods  presented  there  are  un- 
doubtedly not  the  only  ways  that  may  be  developed  but,  at  least  on  the 
basis  of  preliminary  results,  they  would  seem  to  offer  considerable  promise 
for  the  detection  and  correction  of  nonsampling  errors.  This  is  especially 
true  because  the  results  indicate  the  methods  to  be  most  efficient  in  detect- 
ing nonreporters,  the  principal  source  of  nonsampling  errors  in  most  cases. 

Concluding  Comments 

If  the  findings  of  this  project  are  any  indication,  increasing  attention 
must  be  given  to  the  detection  and  correction  of  nonsampling  errors.  Such 
attention  will  be  needed  particularly  in  the  conduct  of  large-scale,  well- 
designed  probability  samples,  for  as  the  efficiency  of  a  sample  design  in- 
creases and  the  size  of  sampling  variances  decreases,  the  effect  of  non- 
sampling  errors  becomes  progressively  more  important.  Since  nonsampling 
variances  are  virtually  unaffected  by  sample  size,  we  are  faced  with  the 
paradoxical  situation  that  the  more  efficient  is  the  sample  design,  the  more 
important  are  nonsampling  errors  likely  to  be  and  the  more  meaningless 
are  confidence  interval  computations  based  on  the  usual  error  formulas. 

To  be  sure,  the  relative  importance  of  nonsampling  errors  will  vary 
with  the  nature  and  subject  matter  of  the  particular  survey.  Yet,  there  is 
little  reason  to  doubt  that  on  many  types  of  surveys,  especially  those  on 
which  numerical  data  are  being  collected,  the  effect  of  such  errors  will  be 
substantial  and  may  seriously  distort  estimates  obtained  by  failing  to  allow 
for  them. 

Under  the  circumstances,  much  future  work  on  survey  methods  needs 
to  be  directed  toward  means  of  coping  with  such  nonsampling  errors.  In 
this  respect,  attention  should  be  given,  not  only  to  the  improvement  of 
survey  procedures,  but  also  to  the  use  of  external  information  in  conjunc- 
tion with  survey  data,  as  has  already  been  discussed.  Indeed,  research  on 
behavioral  science  surveys  has  led  to  the  proposal  that  virtually  every  such 
survey  be  accompanied  by  a  validation  study  to  serve  as  a  basis  for  de- 
tection and  correction  of  nonsampling  errors.160 

100  W.  G.  Madow,  "On  Some  Aspects  of  Response  Error  Measurement,"  Pro- 
ceedings of  the  Social  Statistics  Section  of  the  American  Statistical  Association 
(Washington:   American  Statistical  Association,  1965),  pp.  182-92. 
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To  many  people,  this  proposal  will  seem  extreme,  but  the  fact  remains 
that  until  such  validation  surveys  are  incorporated  in  survey  designs  there 
is  no  way  of  gauging  the  true  extent  of  nonsampling  errors.  To  be  sure, 
such  validation  surveys  will  add  considerably  to  the  cost  of  a  survey  opera- 
tion, but  the  result  in  terms  of  efficiency  may  well  be  worth  this  additional 
cost. 
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BRIEF  ADVANCE  LETTER,    WAVE  1,    STUDY  PI 


Dear  Mr. 

The  purpose  of  this  letter  is  to  ask  for  your  cooperation  in  a 
nationwide  study  on  how  people  handle  their  finances.     Possibly 
you  have  read  or  heard  about  the  operations  of  this  project  in 
other  cities.     If  not,    the  enclosed  copy  of  an  editorial  from 
the  New  York  Times  will  provide  you  with  a  general  description. 

The  study  is  sponsored  by  our  Committee  and  other  nonprofit 
organizations.     It  has  no  connection  -with  any  business  firms 
or  government  organizations,    as  Mr.  


Chancellor  of University,    can  affirm. 

A.  major  aspect  of  this  work  is  being  carried  out  with  a  cross- 
section  of  families  in  the area,    through  interviews 

with  residents  at  addresses  selected  by  statistical  sampling 
from  the  population  at  large.     Your  name  was  obtained  as  the 
resident  of  one  of  the  addresses  we  chose.     Because  you  have 
been  selected  in  this  manner,    we  are  most  anxious  to  secure 
your  cooperation. 

Within  a  few  days,    Mr.  of  our  staff  will 

call  on  you.  The  information  you  give  will  be  kept  in  the 
strictest  confidence.  We  hope  that  we  may  count  on  your 
support. 

Sincerely  yours, 


Robert  Ferber 

Enc. 

271 
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LONGER  ADVANCE  LETTER,    WAVE  1,    STUDY  PI 


Dear  Mr. 

The  purpose  of  this  letter  is  to  ask  for  your  cooperation  in  a  nationwide 
study  on  how  people  handle  their  finances.     Possibly  you  have  read  or 
heard  about  the  operations  of  this  project  in  other  cities.     If  not,    the 
enclosed  copy  of  an  editorial  from  the  New  York  Times  will  provide  you 
with  a  general  description. 

Essentially  the  aim  of  the  project  is  to  gain  a  better  understanding  of  the 
manner  in  which  changes  in  people's  finances  affect  the  ups  and  downs  of 
the  economy.      The  absence  of  such  information  constitutes  one  of  the 
largest  gaps  in  our  present  economic  knowledge.     If  such  information 
were  available,    it  would  constitute  a  major  step  forward  in  the  control 
of  business  fluctuations. 

To  obtain  the  necessary  information  is  a  primary  objective  of  this  study, 
which  is  sponsored  by  our  Committee  and  other  nonprofit  organizations. 
The  study  has  no  connection  with  any  business  firms  or  government 

organizations,    as  Mr.  ,    Chancellor  of 

University,    can  affirm. 

A  major  aspect  of  this  work  is  being  carried  out  with  a  cross-section  of 
families  in  the area,    through  interviews  with  residents  at  ad- 
dresses selected  by  statistical  sampling  from  the  population  at  large. 
Your  name  was  obtained  as  the  resident  of  one  of  the  addresses  we  chose. 
Because  you  have  been  selected  in  this  manner,    we  are  most-anxious  to 
secure  your  cooperation. 

Within  a  few  days,    Mr.  of  our  staff  will  call  on  you. 

The  information  you  give  will  be  kept  in  the  strictest  confidence.     We 
hope  that  we  may  count  on  your  support. 

Sincerely  yours, 


Robert  Ferber 
Enc. 
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ADVANCE  LETTER  WITH  STRAIGHTFORWARD  APPEAL 
WAVE  1,    STUDY  P2 


Dear  Mr. 

I  am  writing  you  on  behalf  of  our  Inter -University  Committee  to  request 
your  assistance  in  a  nationwide  study  we  are  conducting  on  how  people 
handle  their  finances.     Possibly  you  have  read  newspaper  accounts  of 
this  study. 

This  research  study  is  sponsored  by  our  Committee  with  the  assistance 
of  the  Ford  Foundation  and  other  nonprofit  organizations.     It  has  no 
connection  with  any  business  firms  or  government  organizations,    as  you 

may  verify  by  contacting  the Better  Business  Bureau  or  the 

office  of  Dr.  ,    President  of University. 

The  study  is  conducted  by  interviewing  families  in  various  parts  of  the 
country  to  obtain  information  on  their  attitudes  toward  different  forms 
of  saving  and  on  their  money-handling  practices.     The  families  inter- 
viewed are  selected  by  probability  methods  from  lists  of  the  population 
in  the  area. 

Your  name  happens  to  be  one  of  those  selected    Because  you  have  been 
selected  in  this  manner  we  are  most  anxious  to  secure  your  cooperation. 

Within  a  few  days,    Mrs.  ,    who  is  assisting  us  in  this 

study,    will  call  on  you  to  obtain  some  of  the  needed  data.     The  informa- 
tion you  give  will  be  kept  in  the  strictest  confidence;  published  results 
of  the  study  relate  to  groups  of  families,    never  to  an  individual  family. 

We  hope  that  we  may  count  on  your    support. 

Sincerely, 


Robert  Ferber 
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ADVANCE  LETTER  SEEKING  RESPONDENT  SUGGESTIONS 
WAVE  1,    STUDY  P2 


Dear  Mr. 

I  am  writing  you  on  behalf  of  our  Inter- University  Committee  to  request 

your  assistance  in  a  nationwide  study  we  are  conducting  on  how  people  handle 

their  finances.      Possibly  you  have  read  newspaper  accounts  of  this  study. 

This  research  study  is  sponsored  by  our  Committee  -with  the  assistance  of 
the  Ford  Foundation  and  other  nonprofit  organizations.     It  has  no  connection 
with  any  business  firms  or  government  organizations,    as  you  may  verify 

by  contacting  the Better  Business  Bureau  or  the  office  of  Dr. 

,    President  of University. 

The  study  is  conducted  by  interviewing  families  in  various  parts  of  the 
country  to  obtain  information  on  their  attitudes  toward  different  forms  of 
saving  and  on  their  money-handling  practices.      The  families  interviewed 
are  selected  by  probability  methods  from  lists  of  the  population  in  the  area. 

Your  name  happens  to  be  one  of  those  selected.     Because  you  have  been 
selected  in  this  manner  we  are  most  anxious  to  secure  your  cooperation. 
The  nature  of  our  request  is  rather  unusual  in  that  we  seek  not  only  informa- 
tion but  also  advice.     A  primary  aim  of  this  study  is  to  evaluate  and  im- 
prove current  methods  of  obtaining  such  data  from  people  like  yourself. 
Accordingly,    we  should  like  to  have  your  help  in  advising  us  how  well  you 
feel  that  our  questions  succeed  in  securing  the  information.     Also  we  should 
like  to  have  your  suggestions  on  what  improvements  might  be  considered 
in  the  questions  and  interviewing  approach. 

Within  a  few  days,    Mrs.  ,    who  is  assisting  us  in  this  study, 

will  call  on  you  to  obtain  some  of  the  needed  data.     The  information  you  give 
will  be  kept  in  the  strictest  confidence;  published  results  of  the  study  relate 
to  groups  of  families,    never  to  an  individual  family. 

We  hope  that  we  may  count  on  your  support. 

Sincerely, 


Robert  Ferber 
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Page  2  --  IRF 

CLASSIFICATION  DATA  (Fill  in  as  complete  as  possible,    even  for  refusals  and 
non-  contacts) 

Type  of  dwelling:     (Check  one  and  fill  in  data  below) 

I    I   House  O  Apartment 

Approx.   value  $ Approx.   monthly  rentals  $ 

Condition  of  exterior  Condition  of  interior 


Condition  of  interior 


Approximate  age  of  main  wage  earner 

Occupation  of  main  wage  earner 

Size  of  family 

Nationality 

Race 

3.     ATTITUDE  OF  PANEL  MEMBER 

a.     How  would  you  describe  the  panel  member's  attitude  in  each  of  the  following 
respects:     (Check  below) 

Excel-  Very     Interviewer 

lent  Good      Fair      Poor      poor      comments 

Cooperativeness  DO  L~H  d        


Accuracy  of  infor- 
mation given  (  Exact 
figures  for  items 
requested)  □  D  D  D         D         

Completeness  of  in- 
formation (All  assets 
and  liabilities)  □  □  □  □       .  □         

b.     What  was  respondent's  attitude  toward  the  use  of  records  in  supplying  the  figures 
requested  ? 


c.     Did  you  have  the  impression  at  any  time  that  the  respondent  did  not  know  the 
figures  too  well  which  he  gave  you? 

Yes    Q  No   n  Not  sure    O  Explain 
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Page  3  --  IRF 

d.     Did  you  have  the  impression  at  any  time  that  any  figures  were  deliberately  being 
"doctored"  ? 

Yes    □  No   □  Not  sure    □  Explain 


e.     Did  you  have  the  impression  at  any  time  that  any  figures  were  being  withheld  or 
surpressed? 

Yes    □  No   □  Not  sure   □  Explain 

4.     GIFT  INFORMATION 

"Was  a  gift  offered  to  the  respondent?        Yes    LJ  No    [J 

I 
If  Yes:    "What  was  the  respondent's  reaction  to  the  gift?    (Exact  words  of  re- 


spondent^ 


If  No:        Was  copy  of  report  offered  to  respondent  ?      Yes    \~\  No    Q 

If  Yes:    What  was  respondent's  reaction?    (Exact  words,    if  possible) 


SUMMARY 

Please  summarize  briefly  the  overall  reaction  of  the  respondent  to  the  interview. 
Include  any  information  or  observations  which  might  be  of  interest. 
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b.  RECORDS  CONSULTED? 


A.  Life  insurance 

B.  Annuities 

C.  Government  bonds  and  notes 

D.  Other  bonds 

E.  Businesses 

F.  Stocks  or  mutual  funds 
C.   Investment  clubs 

H.   Pension  plans 
I.  Real  estate 
J.  Mortgages  on  real  estate  owned 


Acquired  or 
disposed  of 
No       Yes 

□    cu 

Rec 

cons 
Yes 

□ 

ords 

i  ui.ted 
No 

□ 

cu 

a 

CU 

□ 

D-> 

□ 

CW 

□ 

o* 

a 

D* 

□ 

n* 

a 

D* 

□ 

a* 

□ 

D* 

□ 

ru 

a 

D- 

□ 

a- 

a 

D* 

□ 

D- 

a 

a-* 

□ 

Ck 

a 

D- 

Owned  during 
period 

Rec 

cons 

ords 
ulted 

K. 

Debts 

□ 

D- 

□ 

D- 

L. 

Checking  account(s) 

□ 

D* 

□ 

D- 

M. 

Savings  account(s) 

□ 

D* 

□ 

rw 

N. 

Loans  and  mortgages  lent 

□ 

D- 

□ 

D* 

O. 

Brokerage  account(s) 

D 

CU 

□ 

D* 

P. 

Personal  trust(s) 

D 

d+ 

□ 

D* 

Please   explain  why  not. 


Please  explain  why  not. 


Comments 
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3.   INTERIM  REPORT  NO.  2 


Did  panel  member  mention  receiving  "  Interim  Report  No,  2"? 

NoQ  YesP 

(Exact  words  of  panel  member) 


4.  MAIL  QUESTIONNAIRE  -  Reason  for  not  returning 

Check  the  following  which  the  panel  member  mentioned  as  being  reasons  why  the 
mail  questionnaire  was  not  returned. 

If  at  the  end  of  the  interview  the  panel  member  has  not  volunteered  any  reasons, 
probe  and  check  the  following  list. 

Do  not  suggest  or  mention  possible  reasons  and  do  not  let  panel  member  see  this 
list.     We  want  their  unbiased  answer. 

□  Did  not  have  enough  time  to  do  so 

□  Questionnaire  was  too  long 

□  Mislaid  the  questionnaire 

O  Could  not  understand  the  questions 

□  Forgot  about  it 

□  Questions  were  too  personal 

□  Didn't  feel  it  was  important 

□  Other  (please  specify) 
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5.   SUMMARY  COMMENTS 

Briefly  summarize  the  respondents  overall  reaction  to  this  interview.  Please  in- 
clude any  information  which  you  feel  may  be  helpful  in  understanding  and  analyzing 
the  data  of  this  panel  member. 


6.   INTERVIEWER'S  NOTES 

This  IRF  will  be  returned  to  you  on  the  next  wave  so  use  this  space  to  record  any 
information  you  think  you  may  need.    Following  are  suggestions. 

Future  contact  arrangements 

Who 

When 


Where_ 
How 


Any  information  which  will  make  your  interview  on  the  next  wave  easier,  touchy 
data  areas  or  subjects,  hobbies  or  other  talking  points,  personal  likes  or  dis- 
likes, etc. 
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b.   How  would  you  describe  his,    or  her,   willinguess  to  use  the  accuracy  card 
and  to  give  an  accuracy  estimate  on  each  of  the  following  holdings? 

Checking      Savings      Brokerage    Loans     Real      Mort.    All  types 
accounts     accounts       accounts       lent       estate    owed       of  debts 

Does  not  own-  •    •  Q  Q  D  D  D  D  D 

Has  not  bought  or 
sold  (acquired  or 


□ 

D 

D 

□ 

□ 

□ 

□ 

Used  records; 
card  not  used'    •   • 

□ 

D 

□ 

□ 

D 

D 

D 

Accuracy  card 
used  willingly    •    • 

□ 

D 

□ 

D 

□ 

□ 

D 

Card  used  re- 
luctantly (reason) 

D 

D 

D 

□ 

□ 

D 

□ 

Refused  to  use 
card  (reason)-    •    • 

□ 

□ 

□ 

□ 

D 

D 

□ 

Reason 

Did  you  have  the  impression  at  any  time  that  the  respondent  did  not  know  any 
particular  figure  too  well  which  he  gave  you  or  withheld  or  suppressed  any? 

□  Yes              DNo                 QNotsure 
Explain  in  detail: 


3.    RESULTS  OF  THE  BANK  REQUEST 

[j  Not  used  for  this  panel  member 

□  Signed  all  requests  willingly 

□  Signed  all  requests  reluctantly 

□  Signed  only  some  requests,    not  all  (reason  below) 

□  Refused  to  sign  any  requests  (reason  below) 

Reason 
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4.   SUMMARY  COMMENTS 

Briefly  summarize  the  respondents  overall  reaction  to  the  interview. 
Please  include  any  information  which  you  feel  may  be  helpful  in  under- 
standing and  analyzing  the  data  of  this  panel  member. 


5.    INTERVIEWER'S  NOTES 

This  IRF  will  be  returned  to  you  on  the  next  wave  so  use  this  space  to 
record  any  information  you  think  you  may  need.     Following  are  sugges- 
tions. 


Future  contact  arrangements 


Who 


When 


Where 
How 


Any  information  which  will  make  your  interview  on  the  next  wave  easier, 
touchy  data  areas  or  subjects,  hobbies  or  other  talking  points,  personal 
likes  or  dislikes,   etc. 
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6.   OFFICE  AND  FIELD  PROBLEMS  AND  COMMENTS 
All  Assets 


Checking  Accounts 


Savings  Accounts 


Life  Insurance 


Annuities 


Government  Bonds 


Non-Government  Bonds 


Own  Business 


Brokerage  Accts 


Stocks 


Personal  Trusts 


Loans  Lent 


Real  Estate 


Mtgs.    Owed 


All  types  of  debts 
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3.   ATTITUDE  OF  PANEL  MEMBER 

a.  In  your  opinion,   how  accurate  and  how  complete  is  the  information  that  was 
given  for  each  of  the  following  holdings. 


Checking     Savings      Life      Gov. 


Fully  so.  .  . 
Fairly  so.  . 
Not  much.  . 
Very  little. 


accounts 

accts. 

ins. 

bonds 

bus. 

Stock 

Debts 

□ 

□ 

□ 

□ 

□ 

□ 

D 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

D 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

D 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

□ 

D 

□ 

□ 

□ 

□ 

□ 

□ 

D 

□ 

□ 

□ 

□ 

□ 

D 

□ 

□ 

□ 

□ 

□ 

□ 

Fully  so.  .  . 
Fairly  so.  . 
Not  much.  . 
Very  little. 


Not  applicable □  Q  □  □  □  □  □ 


b.  What  was  respondent's  attitude  toward  the  use  of  records  in  supplying  the  figures 
requested? 


c.  Did  you  have  the  impression  at  any  time  that  the  respondent  did  not  know  the 

figures  too  well  which  he  gave  you?     ,— .  ,,         —          _                                    . 
5  &  /  r-j  Yes   □  No  □  Not  sure   Explain 


d.  Did  you  have  the  impression  at  any  time  that  any  figures  were  deliberately  being 
"doctored"? 


□  Yes    QNo   □  Not  sure      Explain 


e.  Did  you  have  the  impression  at  any  time  that  any  figures  were  being  withheld  or 
suppressed?  Q  Yes    □  No  □  Not  sure       Explain 


f.  Did  you  leave  an  interview  evaluation  form  with  the  panel  member? 

□  Yes    DNo   □  Not  applicable 

What  was  the  respondents  reaction  ? 
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4.      SUMMARY  COMMENTS 

Briefly  summarize  the  respondents  overall  reaction  to  the  interview. 
Please  include  any  information  which  you  feel  may  be  helpful  in  under- 
standing and  analyzing  the  data  of  this  panel  member. 


5.      INTERVIEWER'S  NOTES 

This  IRF  will  be  returned  to  you  on  the  next  wave  so  use  this   space  to 
record  any  information  you  think  you  may  need.     Following  are  sugges- 
tions. 


Future  contact  arrangements 

Who 

When 


Where 


Any  information  which  will  make  your  interview  on  the  next  wave  easier, 
touchy  data  areas  or  subjects,   hobbies  or  other  talking  points,   personal 
likes  or  dislikes,    etc. 
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2. 

Which  of  the  statements  on  this  card  (E)  best  reflects  your  attitude  on  the 

uses  to  which  savings  should  be  put?     (Check  one)  _. 

Dis- 

Best    agree 

A.  People  should  try  never  to  draw  on  savings;  if  extra  money  is 

needed,   they  should  borrow LJ  LJ 

B.  People  should  draw  on  savings  only  in  an    emergency, LJ  C3 

C.  People  should  draw  on  savings  only  in  an  emergency  or  to 

buy  or  build  a  house. LJ  LJ 

D.  People  should  draw  on  savings  not  only  for  the  above  pur- 
poses but  also  for  any  large  expenditures,  such  as  a  car, 
appliances,   take  a  trip,   etc. Q]  Q 

E.  People  should  draw  on  savings  as  needed  when  expenses 

exceed  income. Lj  O 

Comments: 


b.  Now,  which  cf  these  statements  do  you  disagree  with  most?     (Check  one  above. ) 

II  DEBTS 
.a.  Following  is  a  list  of  reasons  why  people  have  debts.     Would  you  tell  me  if  you 
or  a  member  of  your  family  presently  have  a  debt  for  any  of  these  reasons.  Please 
include  everything  no  matter  how  small.     (Do  not  include  bills  payable  in  30  days.  ) 

Do  you  or  any  members  of  your  family  have  any  debts. . . 

Presently        Check  if 

have        records  handled 
Yes        No      outside  family 

A.  Mortgage(s)  on  own  house Q  Q  Q 

B.  Mortgage(s)  on  other  property. □  Q  Q] 

C.  To  repair  or  improve  house  (loan  or  mortgage). O  D  O 

D.  To  buy  a  car D  □  D 

E.  To  buy  appliances,  clothing,   or  other  goods. Qj 

F.  Personal  loan  with  car  or  other  goods  as  collateral Q  L"] 

G.  To  make  a  vacation  trip Q]  Q 

H.   To  pay  medical,   dental  or  other  bills []  O 

I.  Personal  loan  on  insurance,   stock  or  bank  accounts  or 

on  any  other  collateral.. [ 

J.    Refinance  old  loans [   i  l 

K.   Other  loans  (give  purpose  of  loan) CH 

[~l  No  debts  at  present. 
b,    (Ask,   if  they  presently  have  debts.  ) 

Are  the  records  of  any  of  the  debts  you  have  handled  by  someone  outside  your 
family?     (Check  above. ) 
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3- 

c.   (Ask  only  if  no  debt  at  present. ) 
i.  Have  you  or  a  member  of  your  family  ever  borrowed  any  money? 

Yes  □  No 


9 


Under  what  circumstances  (Why?  When?  ) 


ii.  Have  you  or  a  member  of  your  family  ever  bought  anything  on  installment? 
Yes           □   No 
Under  what  circumstances  (Why?  When?  J 


? 


(Skip  to 

Page  5 
We  would  like  to  have  some  details  about  each  of  the  debts  you  mentioned.  Before 
I  start,  let  me  assure  you  once  again  that  the  information  we  obtain  on  these  inter- 
views is  kept  strictly  confidential.     Your  name  and  address  do  not  even  appear  on 
this  questionnaire --only  a  number.     The  code  which  connects  your  number  with  your 
name  and  address  is  kept  under  lock  and  key  at  the  University  of  Illinois  in  Urbana. 


Now,  let  us  take  your  . . . 
(Be  sure  to  cover  every  debt  checked  "yes"  in  Question    £a. ) 
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II.  ASSETS  5. 

,  a.   Would  you  look  over  the  statements  on  this  card  (A)  and  tell  me  which  one,  if 

any,   best  represents  your  attitude  toward  saving.  (Check  one)  Dia- 

Best  agree 

A.  It  is  not  particularly  necessary  to  save  because  if  times  get  bad, 

things  will  work  out  somehow □  ' 

B.  It  is  not  particularly  necessary  to  save  because  we  are  adequately 
covered  by  health  and  accident  insurance  and  in  various  other 

ways • « LJ  LJ 

C.  We  should  save  primarily  for  old  age,  with  perhaps  a  little  in  the 
bank  for  emergencies LJ 

D.  We  should  save  not  only  for  old  age  but  saving  is  also  necessary 

for  many  other  reasons LJ  LJ 

E.  We  give  priority  to  saving  under  almost  any  circumstances □  LJ 

b.  Now,  which  of  these  statements  do  you  disagree  with  most  strongly? 
(Check  one  box  in  "disagree"  column.) 
Comments: 


Would  you  look  over  these  statements  (Card  B)  on  the  different  ways  in  which 
people  handle  their  income  and  tell  me  which,  if  any  of  them,   best  describes 
your  practices?   (Circle  one.) 

A.  Buy  whatever  you  need  as  long  as   you  can  get  the  credit. 

B.  Try  to  keep  your  spending  just  within  your  income. 

C.  Put  aside  whatever  is  left  over  at  the  end  of  the  week  (or  month). 

D.  Try  to  keep  expenditures  down  so  that  some  money  is  left  over  for 
saving  at  end  of  week  (or  month,   or  year). 

E.  Have  definite  plan  in  advance  for  saving  part  of  income,   as  through 
fixed  savings  plans,   etc, ,   and  then  spend  remainder. 

F.  Not  only  have  definite  plan  in  advance  for  saving  part  of  income,  but  also 
try  to  have  something  left  over  from  expenses  for  savings,  too. 

Comments:  _____ 


If  none,  please  explain_ 
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II  ASSETS 

9.  a.  Following  is  a  list  of  asset  holdings.  Would  you  please  tell  me  which  ones  you 
or  any  member  of  your  family  have.  Include  everything,  no  matter  how  small. 
Do  you  or  any  members  of  your  family  have  any  . . . 

Regarded  Check  if 

Presently  have  as  savings  records  handled 

Yes  No  Yes        No  outside  family 

l~l  A.   Checking  accounts  D        D 

l~l   B.  Savings  accounts --banks,    savings  O        D 

and  loan  associations,   credit 
unions,  postal  savings,   or  other 
savings  accounts. 

QC.     Owner  or  part-owner  of  anunincor-        D        d  i 

porated  business. 

Qj  D.     Owner  or  part-owner  of  an  incor-  D        D  D 

porated  business  and  active  in  its 
management. 

LjE.     Life  insurance --ordinary  life,   en-  Q        Q  Q 

dowment,   term,   children's  educa- 
tion,  group.   Gov't,  mortgage,   cre- 
dit,  savings,   business  life  or  other. 

QF.     Annuities  O        D  □ 

II  G.     Pension  plans  O        O  ! 

DH.     U.   S.   Treasury  notes  D  D  D 

(7]l  •     Savings  bonds --series  E,    series  H,  Q  iZl  D 

and  other. 

DJ.     Public  authority  bonds --state,  mun-  D  CD  D 

icipal,    toll  -road,   etc. 

QK.     Corporate  bonds  or  debentures.  D  D  Q 

Ql,     Other  bonds --church,   foreign,  etc.  Q  Q  O 

rl  M.     Common  or  preferred  stock,    incl.  D        Q  O 

mutual  funds. 
Qn.      Brokerage  accounts  l~l        l~l 


n 

D 

D 

D 

D 

D 

□ 

D 

□ 

□ 

□ 

G 

□ 

D 

n 

D 

□ 

□ 

□ 

□ 

□  O.     Own   your  home  DO  O 

D  P-     Own  land  D        D  Q 

QQ.     Own  other  real  estate  D        D  Q 

DR.     Mortgages  lent  D        D  Q 

QS.     Loans  to  relatives,   friends  or  Q        D  Q 

business  associates.  I~l        t~l  [~1 
l~)   No  holdings  in  any  of  the  above  forms.   (Skip  to  6d) 

b.  Which  of  these  holdings  that  you  have  do  you  regard  as  savings?     (Record  above) 
Comments 


Are  the  records  of  any  of  the  holdings  that  you  have  handled  by  someone  outside 
your  family  ?     (Check  above. ) 
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d.  Listed  on  this  card  (C)  are  some  of  the  goals  which  people  have  in  building  up 
savings.     Which  if  any  of  these  represent  your  goals? 

□  To  provide  for  our  old  age. 

Q   To  provide  an  inheritance  for  our  children. 
C3  To  provide  for  our    childrens'  education. 

□  To  pay  off  debts. 

n  To  buy  or  build  a  house. 

O   To  purchase  some  particular  item,    such  as  a  car,  furniture,   appliance, 
etc. 

O   To  make  a  trip  at  some  future  time,    such  as  a  vacation. 

n  For  emergencies  or  a  rainy  day. 

Comments: 


Do  you  have  any  savings  goals  which  are  not  listed  on  this  card? 
10. 


We  would  like  to  have  some  details  about  each  of  the  holdings  you  mentioned. 
Before  1  start,  let  me  assure  you  once  again  that  the  information  we  obtain  on 
these  interviews  is  kept  strictly  confidential.     Your  name  and  address  do  not  even 
appear  on  this  questionnaire --only  a  number.     The  code  which  connects  your 
number  with  your  name  and  address  is  kept  under  lock  and  key  at  the  University 
of  Illinois  in  Urbana, 

Now  let  us  take  your 

(Make  sure  to  cover  every  item  checked  at  the  top  of  pp.  5-10. ) 
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IV  INCOME 


15. 


,  a.   Which  family  members  presently  have  jobs  from  which  they  receive  salaries,  wages 
or  commissions?     Include  not  only  full  time  work,  but   also  part-time,  temporary  and 
summer  jobs  (baby  sitting,   newspaper  routes,    solicitation  work,   consulting,   etc.). 
(If  any  family  member  had  income  from  more  than  one  job,  use  one    column  for  each 
job  and  note  whether  it  is  that  person 's  1st,   2nd,   3rd,   etc.,  job.). 

Relation  to  main 
wage  earner 
Job  number 
Time  employed: 

Full  time 

Part-time 

Temporary 

Summer 

Other  (explain) 
Type  of  work 

b.   Would  you  look  over  the  sources  of  income  on  this  Card  (F)  and  tell  me  from  which 
you  have  or  expect  to  receive  income  this  year,   1958,   (Check  appropriate  boxes. ) 


MWE 

□ 
□ 
□ 
□ 

□ 
□ 

□     ■ 
□ 

□ 
□ 
□ 

□ 
□ 
□ 
□ 

□ 
□ 
□ 
□ 

1.  Net  income  from  business 

2.  Interest  or  dividends  from  savings 

accounts. 

3.  Interest  on  Federal  Government 

bonds 

4.  Interest  on  other  bonds. 

5.  Dividends  on  stocks. 

6.  Profit  from  rents  collected 

7.  Bonuses 

8.  Retirement  pay 

9.  Unemployment  or  workmen's  com- 

pensation. 
10.  Old  age  pension 


11.  Social  security  I 

12.  Welfare  or  relief  payments  □ 

13.  Inheritances  or  bequests  !  J 

14.  Income  tax  refunds  received  LJ 

15.  Insurance  settlements  I    i 

16.  Trust  funds  □ 

17.  Capital  gains  [  ] 

18.  Royalties  D 

19.  Alimony  □ 

20.  Veterans  pensions  and 

allotments  M 

21 .  Annuities  [J 

22.  Gifts  or  prizes  (value)  LJ 

23.  Gambling  winnings  Q 

24.  Other  sources  l~l 


Taking  into  consideration  all  income  of  wages  and   salaries  plus  income 
from  these  other  sources,   in  what  range  do  you  expect  your  family's  total 
income  to  fall  for  this  year,   1958.   (Card  G) 


A. 

Less  than  $1,000 

□ 

B. 

$1,000  -      $2,999 

n 

C. 

$3,009-      $4,999 

n 

D. 

$5,000-      $7,499 

n 

E. 

$7,500-     $9,999 

□ 

F. 

$  10,000 

$  14,999 

□ 

G. 

$  15,000 

$  24,999 

□ 

H. 

$  25,000 

$  49,999 

□ 

I. 

$  50,000 

$  99,999 

□ 

J. 

$100.  000 

$249,999 

□ 

K. 

$250,000 

or  more 

□ 

d.  How  do  you  expect  your  family's  income  this  year  to  compare  to  last  year? 
Up  %  No  change  □  Down % 
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V.   TERMINATING  INTERVIEW  16. 

1.  Asset  and/ or  debt  records  held  by  other  person. 

(If  any  of  the  holdings  listed  in  Question  6a  or  question  10a  are  handled  by  a  person 
outside  the  family,  i.e. ,  if  there  is  a  check  in  the  box  marked  "records  handled 
outside  family",   ask  the  following  question): 

a.  Who  handled  these  holdings?    Assets. 

Debts   '•  

b.  We  would  like  to  get  some  details  about  these  holdings.     Could  we  have  your 

permission  to  go  to (identity  of  other  person)  and  obtain 

these  details  from  him?     Let  me  assure  you  once  again  that  the  information  we 
obtain  on  these  interviews  is  kept  strictly  confidential.     Your  name  and  address 

do  not  even  appear  on  this  questionnaire--only  a  number.     The  code  which  connects 
your  number  with  your  name  and  address  is  kept  under  lock  and  key  at  the  University 
of  Illinois  in  Urbana. 

Willing  □ 

Arrangements  for  contact. 


Not  willing  □   Why? 


.   Thank  respondent(s)  for  interview.    Give  assurances  of  confidentiality  once  more. 

3.  Offer  gift,  if  respondent  eligible. 

(Should  be  offered  only  to  respondents  who  meet  both  of  the  following  require- 
ments.    (If  information  on  either  of  these  points  has  been  refused,  use  your 
judgment  as  to  whether  or  not  to  offer  gift. ) 

1.  Family  head  is  not  a  college  graduate. 

2.  Total  family  income  is  not  over  $7,500.) 

4.  Offer  to  send  copies  of  survey  report,  if  judged  useful. 

(Whether  or  not  respondent  is  offered  gift,  he  may  be  offered  copies  of  releases 
and  reports  on  this  study  as  soon  aa  they  are  completed.) 

5.  Sound  out  respondent(a)  on  future  cooperation 

(Respondents  should  be  told  that  this  is  a  continuing  study  and  information  will 
be  gathered  not  more  than  4  times  a  year  for  a  period  of  not  more  than  2  years. 
It  should  also  be  stressed  that  future  questionnaires  will  be  shorter  inasmuch 
as  background  information  has  already  been  furnished.) 
a.     Re spondent( s )  queried:     YesQ     No  Q-»Why? 


b.    Method  of  future  cooperation  suggested. 

□  Mail 

n   Personal  interview  at  home 

J   Personal  interview  at  office 
O   Other  ^^ 


6.     Thank  respondent(s)  again. 
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Any  owned  jtly? 
5a.    How  much    of  each  of  the  following  What  is  their  total  (If  yes)  What 

types  of  livestock  do  yoa  own?  (Card  A)  How  many?  current  value?  pet  ia  yours? 

E|  $ El 

El 


Any  others?   (fill  in  above)     |N] 

b.    Whataboutmachines,  including  attach  - 
ments?     (Card  B) 


El 


El 

.0 


.El 

M 
M 
El 
_E 
El 
.El 
El 


Any  others?   (fill  in  above) 
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Table  1.  Distribution  of  Cooperativeness  Ratings,  by  Respondent 
Characteristics,,  Study  PI,  Wave  1 


Characteristic 


Category- 


Base 
number 


Percentage  of  total 
respondents  rated 


Excel- 
lent 


Good 


Fair 


Poor 

or  very 

poor 


Value  of  home . 


Age  of  main  wage  earner . 


Size  of  savings  unit. 


Nationality . 
Occupation , 


Education . 


Income  of  savings  unit. 


Gross  value  of  assets. 


Amount  in  savings  accounts . 


Renter 

Under  $10,000 
$10,000  to  $24,999 
$25,000  to  $49,999 
$50,000  or  more 

Under  35 
36  to  49 
50  to  64 
65  or  over 

1 
2 
3 
4 
5  or  more 

United  States 
Other 

Professional- 
manager 
Self-employed 
Clerical-sales 
Laborer 
Retired 
Other 

Grade  school 
High  school 
College  or  more 

Under  $5,000 
$5,000  to  $9,999 
$10,000  to  $24,999 
$25,000  or  more 

Under  $10,000 
$10,000  to  $49,999 
$50,000  to  $99,999 
$100,000  or  more 

None 

Under  $100 
$100  to  $999 
$1,000  to  $4,999 
$5,000  or  more 


46 

8 

67 

68 

9 

33 
54 
89 
23 

24 
60 
38 
41 
36 

186 

13 

70 

40 
40 
19 
14 
14 

28 

54 

113 

30 
69 
63 
24 

19 
61 
36 
47 

27 
26 
47 
32 
28 


39 
38 
61 
51 
67 

49 
54 
53 
49 

42 
45 
40 
66 
67 

52 
54 

57 

42 
50 
68 
57 
36 

54 
44 
57 

40 
57 
57 
62 

42 
57 
61 
66 

44 
61 
57 
62 
68 


43 
38 
25 
21 
21 

33 
30 
29 
17 

33 
30 
40 

17 
25 

27 
46 

26 

30 
39 

21 
14 
43 

32 
31 
24 

50 
28 
29 

17 

47 
31 
31 
19 

37 
23 
30 
28 
25 


17 
12 

4 
19 

0 

12 
13 
10 
22 

21 
13 
13 
12 
5 

13 
0 

11 

12 
10 
10 

21 
21 

4 
20 
11 

10 

12 

17 

8 

10 

10 

6 

11 

7 

11 

13 

9 

7 


0 
12 

9 

9 

11 

6 

4 

8 

13 

4 
12 
8 
5 
3 

8 
0 

6 

15 

2 
0 
7 
0 

11 
4 
8 

0 

4 

8 

12 

0 
2 
3 

4 

11 
4 
0 
0 
0 
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Table  2.  Distribution  of  Accuracy  Ratings,  by  Respondent 
Characteristics,,  Study  PI,  Wave  1 


Characteristic 


Category 


Base 
number 


Percentage  of  total 
respondents  rated 


Excel- 
lent 


Good 

Fair 

29 

24 

38 

12 

36 

12 

17 

23 

21 

7 

38 

19 

26 

19 

28 

16 

22 

22 

29 

17 

22 

17 

35 

24 

20 

18 

41 

12 

29 

18 

46 

15 

27 

23 

24 

13 

42 

10 

32 

16 

35 

7 

29 

21 

36 

7 

24 

24 

27 

17 

30 

17 

33 

22 

26 

14 

35 

13 

35 

15 

40 

18 

28 

17 

23 

18 

35 

13 

44 

12 

33 

13 

16 

26 

25 

21 

Poor 

or  very 

poor 


Value  of  home . 


Age  of  main  wage  earner. 


Size  of  savings  unit. 


Nationality . 
Occupation , 


Education . 


Income  of  savings  unit. 


Gross  value  of  assets. 


Amount  in  savings  accounts . 


Renter 

Under  $10,000 
$10,000  to  $24,999 
$25,000  to  $49,999 
$50,000  or  more 

Under  35 
36  to  49 
50  to  64 
65  or  over 

1 
2 
3 
4 
5  or  more 

United  States 
Other 

Professional- 
manager 
Self-employed 
Clerical-sales 
Laborer 
Retired 
Other 

Grade  school 
High  school 
College  or  more 

Under  $5,000 
$5,000  to  $9,999 
$10,000  to  $24,999 
$25,000  or  more 

Under  $10,000 
$10,000  to  $49,999 
$50,000  to  $99,999 
$100,000  or  more 

None 

Under  $100 
$100  to  $999 
$1,000  to  $4,999 
$5,000  or  more 


45 

8 

67 

65 

14 

32 
54 
87 
23 

24 
58 
37 
40 
34 

182 

13 

70 

38 
38 
19 
14 
14 

28 

58 

106 

30 
67 
62 
23 

20 
60 
36 
44 

31 
25 
46 
31 
28 


33 
25 
40 
37 
21 

34 
37 
37 
35 

38 
34 
27 
48 
35 

36 
39 

37 

34 
32 
53 
43 
29 

39 
33 
39 

47 
37 
42 
26 

45 
37 
44 
50 

35 
36 
43 
45 
54 


13 
25 
12 
23 
50 

9 
19 
20 
22 

17 
26 
13 
15 
12 

18 
0 

13 

29 
16 
0 
14 
21 

18 
19 
17 


18 
26 

5 

5 

11 

9 

16 

8 

11 

13 

0 
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Table  3.  Distribution  of  Completeness  Ratings,  by  Respondent 
Characteristics,  Study  PI,  Wave  1 


Characteristic 


Value  of  home . 


Age  of  main  wage  earner . 


Size  of  savings  unit. 


Nationality . 
Occupation , 


Education , 


Income  of  savings  unit. 


Gross  value  of  assets. 


Amount  in  savings  accounts . 


Category 


Renter 

Under  $10,000 
$10,000  to  $24,999 
$25,000  to  $49,999 
$50,000  or  more 

Under  35 
36  to  49 
50  to  64 
65  or  over 

1 
2 
3 
4 
5  or  more 

United  States 
Other 

Professional- 
manager 
Self-employed 
Clerical-sales 
Laborer 
Retired 
Other 

Grade  school 
High  school 
College  or  more 

Under  $5,000 
$5,000  to  $9,999 
$10,000  to  $24,999 
$25,000  or  more 

Under  $10,000 
$10,000  to  $49,999 
$50,000  to  $99,999 
$100,000  or  more 

None 

Under  $100 
$100  to  $999 
$1,000  to  $4,999 
$5,000  or  more 


Base 
number 


43 

8 

67 

66 

9 

32 
41 
89 
23 

23 
59 
37 
39 
36 

201 
11 

69 
38 
40 
18 
14 
13 

32 

49 

109 

30 
67 
64 
22 

20 
59 
36 
45 

28 
25 
46 
31 
28 


Percentage  of  total 
respondents  rated 


Excel- 
lent 


33 
13 
46 
35 
33 

38 
29 
35 

30 

35 
32 
24 
49 
47 

34 
27 

36 
32 
38 
67 
36 
23 

34 
43 
37 

37 
43 
39 
32 

35 
46 
44 

44 

39 
36 
39 
48 
46 


Good 

Fair 

40 

21 

50 

12 

34 

10 

23 

21 

33 

0 

34 

9 

39 

17 

33 

19 

26 

22 

43 

13 

27 

17 

35 

30 

26 

18 

36 

3 

28 

15 

55 

18 

30 

19 

32 

13 

40 

13 

17 

17 

29 

14 

46 

31 

38 

16 

24 

20 

33 

14 

50 

10 

31 

18 

28 

17 

36 

14 

40 

25 

41 

12 

31 

14 

33 

13 

32 

11 

40 

12 

39 

20 

26 

19 

36 

14 

Poor 

or  very 

poor 


7 
25 

9 
21 
33 

19 
15 
13 
22 

9 
24 
11 

8 
14 

13 
9 

14 
24 
10 

0 
21 

0 

13 
12 

17 

3 

7 

16 

18 

0 

2 

11 

9 

18 

12 

2 

6 

4 
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Table  4.   Distribution  of  Use-of-Records  Reports,  by  Respondent 
Characteristics,  Study  PI,  Wave  1 


Percentage  of  total 

Characteristic 

Category 

Base 

reporting  use 

number 

Always 

Some- 
times 

Never 

Value  of  home 

Renter 

49 

27 

18 

55 

Under  $10,000 

5 

20 

60 

20 

$10,000  to  $24,999 

66 

35 

36 

29 

$25,000  to  $49,999 

57 

26 

19 

54 

$50,000  or  more 

6 

17 

17 

67 

Age  of  main  wage 

earner 

Under  35 

35 

46 

20 

34 

36  to  49 

50 

34 

26 

40 

50  to  64 

80 

20 

33 

48 

65  or  over 

18 

22 

11 

67 

Size  of  savings  unit.  .  .  . 

1 

24 

17 

25 

58 

2 

57 

21 

30 

49 

3 

35 

17 

34 

49 

4 

25 

36 

28 

36 

5  or  more 

23 

13 

26 

61 

Nationality 

United  States 

171 

30 

28 

43 

Other 

12 

17 

8 

75 

Occupation 

Professional- 

manager 

62 

32 

31 

37 

Self-employed 

39 

18 

8 

74 

Clerical-sales 

41 

27 

41 

32 

Laborer 

18 

56 

11 

33 

Retired 

9 

22 

22 

56 

Other 

14 

21 

36 

43 

Education 

Grade  school 

30 

33 

30 

37 

High  school 

48 

23 

31 

46 

College  or  more 

102 

31 

23 

45 

Income  of  savings  unit 

Under  $5,000 

31 

19 

35 

45 

$5,000  to  $9,999 

66 

41 

26 

33 

$10,000  to  $24,999 

60 

28 

27 

45 

$25,000  or  more 

19 

16 

21 

63 

Gross  value  of  assets .  .  . 

Under  $10,000 

21 

29 

19 

52 

$10,000  to  $49,999 

62 

39 

34 

27 

$50,000  to  $99,999 

31 

19 

39 

42 

$100,000  or  more 

43 

37 

21 

42 

Amount  in  savings 

accounts 

None 

30 

37 

13 

50 

Under  $100 

25 

36 

40 

24 

$100  to  $999 

46 

35 

30 

35 

$1,000  to  $4,999 

29 

17 

38 

45 

$5,000  or  more 

28 

39 

25 

36 
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Table  5.  Distribution  of  Evidence  of  Withholding  Data, 
by  Respondent  Characteristics,  Study  PI,  Wave  1 


Percentage  of  total 

B 

withholding  data 

Characteristic 

Category 

number 

No 

Not 

Yes 

sure 

Value  of  home 

Renter 

Under  $10,000 

47 
8 

70 
50 

17 
25 

13 

25 

$10,000  to  $24,999 

68 

82 

1 

16 

$25,000  to  $49,999 

68 

60 

10 

29 

$50,000  or  more 

9 

67 

11 

22 

Age  of  main  wage  earner .  .  . 

Under  35 

33 

76 

15 

9 

36  to  49 

55 

78 

5 

16 

50  to  64 

90 

64 

11 

24 

65  or  over 

23 

61 

4 

35 

Size  of  savings  unit 

1 

25 

72 

12 

16 

2 

60 

62 

12 

27 

3 

38 

63 

8 

29 

4 

42 

79 

10 

12 

5  or  more 

36 

78 

6 

17 

Nationality 

United  States 

188 

69 

10 

21 

Other 

13 

77 

0 

23 

Occupation 

Professional- 

manager 

71 

77 

6 

17 

Self-employed 

39 

54 

10 

36 

Clerical-sales 

41 

76 

10 

15 

Laborer 

19 

89 

11 

0 

Retired 

14 

50 

7 

43 

Other 

15 

53 

27 

20 

Education 

Grade  school 

28 

71 

11 

18 

High  school 

54 

67 

11 

22 

College  or  more 

114 

72 

7 

21 

Income  of  savings  unit 

Under  $5,000 

31 

77 

19 

3 

$5,000  to  $9,999 

70 

81 

7 

11 

$10,000  to  $24,999 

63 

70 

6 

24 

$25,000  or  more 

24 

58 

17 

25 

Gross  value  of  assets 

Under  $10,000 

20 

75 

25 

0 

$10,000  to  $49,999 

62 

90 

3 

6 

$50,000  to  $99,999 

36 

67 

11 

22 

$100,000  or  more 

48 

75 

10 

15 

Amount  in  savings  accounts 

None 

28 

68 

11 

21 

Under  $100 

24 

71 

12 

17 

$100  to  $999 

48 

85 

4 

10 

$1,000  to  $4,999 

33 

79 

15 

6 

$5,000  or  more 

29 

83 

3 

14 
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Table  6.   Distribution  of  Use-of-Records  Ratings,  by  Respondent 
Characteristics,  Study  P2,  Wave  1 


Characteristic 


Percentage  of  total 

Category 

Base 

number 

respondents  rated 

Always  or 

sometimes 

Never 

Renter 

34 

62 

38 

Under  $10,000 

83 

47 

53 

$10,000  to  $24,999 

141 

62 

38 

$25,000  to  $49,999 

31 

52 

48 

$50,000  or  more 

5 

40 

60 

Under  35 

22 

59 

41 

36  to  49 

75 

63 

37 

50  to  64 

108 

57 

43 

65  or  over 

87 

48 

52 

1 

57 

47 

52 

2 

126 

55 

45 

3 

58 

57 

43 

4 

29 

72 

28 

5  or  more 

24 

63 

37 

United  States 

254 

57 

43 

Other 

40 

50 

50 

Professional- 

manager 

63 

56 

44 

Self-employed 

30 

63 

37 

Clerical-sales 

44 

57 

43 

Laborer 

88 

64 

36 

Retired 

63 

44 

56 

Other 

5 

40 

60 

Grade  school 

81 

52 

48 

High  school 

107 

55 

45 

College  or  more 

103 

61 

39 

Under  $5,000 

101 

50 

50 

$5,000  to  $9,999 

106 

66 

34 

$10,000  to  $24,999 

64 

59 

41 

$25,000  or  more 

4 

50 

50 

Under  $10,000 

29 

62 

38 

$10,000  to  $49,999 

137 

65 

35 

$50,000  to  $99,999 

42 

76 

24 

$100,000  or  more 

28 

57 

43 

Under  $100 

19 

63 

37 

$100  to  $999 

70 

51 

49 

$1,000  to  $4,999 

114 

61 

39 

$5,000  or  more 

91 

52 

48 

Value  of  home . 


Age  of  main  wage  earner . 


Size  of  savings  unit . 


Nationality . 
Occupation. 


Education . 


Income  of  savings  unit . 


Gross  value  of  assets . 


Amount  in  savings  accounts. 
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Table  7.  Distribution  of  Willingness-to-Give-Financial-Information  Ratings, 
by  Respondent  Characteristics,,  Study  P2,  Wave  5 


Characteristic 


Category 


Base 
number 


Percentage  of  total 
respondents  rated 


Excel- 
lent 


Good 

Fair 

36 

0 

34 

17 

38 

13 

45 

10 

0 

0 

59 

6 

27 

11 

39 

12 

38 

14 

29 

18 

41 

12 

33 

7 

38 

23 

36 

5 

34 

12 

54 

14 

35 

10 

33 

5 

48 

10 

35 

14 

34 

16 

25 

25 

37 

12 

37 

19 

35 

6 

29 

16 

40 

12 

44 

7 

25 

0 

33 

6 

36 

13 

35 

11 

50 

5 

57 

7 

44 

8 

32 

9 

34 

16 

Poor 

or  very 

poor 


Value  of  home . 


Age  of  main  wage  earner . 


Size  of  savings  unit. 


Nationality . 
Occupation , 


Education 


Income  of  savings  unit. 


Gross  value  of  assets. 


Amount  in  savings  accounts  . 


Renter 

Under  $10,000 
$10,000  to  $24,999 
$25,000  to  $49,999 
$50,000  or  more 

Under  35 
36  to  49 
50  to  64 
65  or  over 

1 
2 
3 
4 
5  or  more 

United  States 
Other 

Professional- 
manager 
Self-employed 
Clerical-sales 
Laborer 
Retired 
Other 

Grade  school 
High  school 
College  or  more 

Under  $5,000 
$5,000  to  $9,999 
$10,000  to  $24,999 
$25,000  or  more 

Under  $10,000 
$10,000  to  $49,999 
$50,000  to  $99,999 
$100,000  or  more 

Under  $100 
$100  to  $999 
$1,000  to  $4,999 
$5,000  or  more 


22 

53 

105 

20 

5 

17 
62 
75 
50 

28 
86 

43 
26 
22 

177 
28 

48 
21 
31 
69 
32 
4 

51 
75 
79 

62 

91 

45 

4 

18 

124 

37 

23 

14 
48 
76 
62 


59 
47 
43 
45 
80 

29 
56 

47 
42 

46 
45 
51 
38 
54 

50 
29 

52 
62 
32 
48 

44 
25 

45 
39 
56 

48 
46 
44 
75 

56 
46 
51 
41 

36 
44 
55 
45 


5 
2 
6 
0 
20 

6 
5 
3 
6 

7 
2 
9 
0 
5 

5 

4 

2 
0 

10 
3 
6 

25 

6 
5 
3 

6 
2 
2 
0 

6 
5 
3 
5 

0 
4 
4 
5 
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Table  8.   Distribution  of  Savings-Account-Accuracy  Ratings, 
by  Respondent  Characteristics,  Study  P2,  Wave  1 


Percentage  of  total 

Category 

Base 
number 

respondents 

rated 

Characteristic 

Not  much 

Fully 

Fairly 

or  very 
little 

Value  of  home 

Renter 

31 

58 

29 

13 

Under  $10,000 

73 

37 

42 

21 

$10,000  to  $24,999 

134 

53 

37 

10 

$25,000  to  $49,999 

30 

47 

33 

20 

$50,000  or  more 

4 

50 

50 

0 

Age  of  main  wage 

earner 

Under  35 

22 

59 

23 

18 

36  to  49 

70 

54 

37 

10 

50  to  64 

100 

52 

37 

11 

65  or  over 

79 

37 

42 

21 

Size  of  savings  unit .  . 

1 

48 

31 

50 

19 

2 

118 

50 

35 

15 

3 

55 

51 

35 

14 

4 

28 

61 

32 

7 

5  or  more 

24 

54 

38 

8 

Nationality 

United  States 

117 

50 

36 

14 

Other 

15 

43 

43 

14 

Occupation 

Professional- 

manager 

61 

56 

38 

7 

Self-employed 

27 

37 

37 

26 

Clerical-sales 

43 

47 

37 

16 

Laborer 

79 

56 

30 

14 

Retired 

58 

38 

43 

19 

Other 

4 

50 

0 

50 

Education 

Grade  school 

72 

40 

39 

21 

High  school 

98 

49 

36 

15 

College  or  more 

99 

55 

36 

9 

Income  of  savings 

unit 

Under  $5,000 

90 

41 

41 

18 

$5,000  to  $9,999 

101 

55 

37 

8 

$10,000  to  $24,999 

62 

53 

34 

13 

$25,000  or  more 

4 

25 

50 

25 

Gross  value  of  assets 

Under  $10,000 

27 

59 

26 

15 

$10,000  to  $49,999 

128 

58 

34 

8 

$50,000  to  $99,999 

39 

54 

41 

5 

$100,000  or  more 

28 

50 

39 

11 

Amount  in  savings 

accounts 

Under  $100 

27 

48 

48 

4 

$100  to  $999 

61 

48 

36 

16 

$1,000  to  $4,999 

94 

54 

37 

9 

$5,000  or  more 

78 

46 

40 

14 
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Table  9.  Distribution  of  Low  Ratings  on  Accuracy 

and  Completeness  of  Checking  Account  Balances, 

by  Respondent  Characteristics,  Study  P3,  Wave  1 


Characteristic 


Category 


Base  number 


Accu- 
racy 


Com- 
pleteness 


Percent  rated 
:'not  much"  or 
'very  little"  on 


Accu- 
racy 


Com- 
pleteness 


Condition  of  home . 


Age  of  farmer . 


Size  of  savings  unit. 


Nationality 

Tenure  status. . .  . 
Education 

Income  of  savings 
unit 


Value  of  assets. 


Size  of  farm . 


Amount  of  farm  debt 


Amount  in  savings 
accounts 


Excellent 
Good  to  excellent 
Fair  to  good 
Poor  to  fair 

Under  35 
35  to  49 
50  to  64 
65  or  over 

1 
2 
3 
4 
5  or  more 

United  States 
Other 

Owner 
Operator  only 

Grade  school 
High  school 
College  or  more 

Under  $2,500 
$2,500  to  $4,999 
$5,000  to  $9,999 
$10,000  to  $14,999 
$15,000  or  more 

Under  $25,000 
$25,000  to  $49,999 
$50,000  to  $99,999 
$100,000  or  more 

Under  100  acres 
100  to  199  acres 
200  to  299  acres 
300  acres  or  more 

Under  $1,000 
$1,000  to  $4,999 
$5,000  to  $9,999 
$10,000  or  more 

None 

Under  $1,000 
$1,000  to  $4,999 
$5,000  or  more 


41 

171 

89 

43 

51 
135 
115 

48 

24 
106 
60 
58 
99 

317 
12 

230 
119 

118 

199 

32 

126 
98 
78 
16 
11 

62 
88 

106 

72 

44 

85 

85 

134 

72 

93 

71 

109 

217 
67 
34 
19 


40 

169 

87 

44 

50 
131 
114 

50 

22 
106 
60 
56 
99 

314 
11 

229 
116 

120 

193 

32 

125 

105 

77 

16 

11 

61 

89 

101 

74 

48 

80 

87 

129 

69 
95 
70 

107 

215 
66 
33 
19 


9 
12 
15 
14 

8 
12 
17 
10 

13 
16 
17 
16 
6 

14 
8 

14 
10 

18 

10 

9 

11 
10 
12 
25 
9 

10 

17 

5 

17 

18 
12 
12 
12 

14 
18 


12 
3 

21 
11 


0 
4 
6 

14 

2 

2 

11 

4 

9 
7 
7 
0 
4 

6 
0 

6 
4 

8 

4 
7 

2 
5 
1 
13 
0 

5 

6 
2 
8 

10 
6 
5 
3 

12 
5 
3 
3 

4 
5 
6 
5 
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Table  10.   Distribution  of  Low  Ratings  on  Figures  Withheld 

and  Lack  of  Knowlegde  of  Figures,  by  Respondent 

Characteristics,,  Study  P3,  Wave  1 


Characteristic 


Category 

Base 
number 

Percentage  rated 
"yes"  on 

Data 

withheld 

Lack  of 
knowledge 

Excellent 
Good  to  excellent 
Fair  to  good 
Poor  to  fair 

42 

175 

96 

46 

5 

9 

7 

13 

2 

18 
15 
20 

Under  35 
35  to  49 
50  to  64 
65  or  over 

51 
139 
122 

52 

2 

4 

13 

13 

10 
10 
18 
31 

1 
2 
3 
4 
5  or  more 

25 

113 

64 

59 

101 

8 
13 
8 
5 
5 

20 
20 
14 
8 
15 

United  States 
Other 

331 
13 

9 
0 

16 
15 

Owner 
Tenant 

240 

124 

11 
3 

19 
9 

Grade  school 
High  school 
College  or  more 

127 

203 

34 

12 

16 

6 

18 

16 

6 

Under  $2,500 
$2,500  to  $4,999 
$5,000  to  $9,999 
$10,000  to  $14,999 
$15,000  or  more 

132 

112 

80 

17 

11 

8 
4 
5 
6 
18 

14 

13 
15 
29 
27 

Under  $25,000 
$25,000  to  $49,999 
$50,000  to  $99,999 
$100,000  or  more 

68 

91 

108 

75 

4 

8 

5 

12 

12 
18 
14 
19 

Under  100  acres 
100  to  199  acres 
200  to  299  acres 
300  acres  or  more 

52 
87 
87 

137 

15 

10 

3 

7 

29 
15 
13 
13 

Under  $1,000 
$1,000  to  $4,999 
$5,000  to  $9,999 
$10,000  or  more 

75 

99 

74 

111 

17 
8 
3 
5 

19 

20 

9 

14 

None 

Under  $1,000 
$1,000  to  $4,999 
$5,000  or  more 

226 

71 
34 
20 

6 

4 

12 

15 

14 
14 
26 
20 

Condition  of  home. 


Age  of  farmer. 


Size  of  savings  unit , 


Nationality 

Tenure  status 

Education 

Income  of  savings  unit 
Value  of  assets 


Size  of  farm . 


Amount  of  farm  debt . 


Amount  in  savings  accounts 


360     The  Reliability  of  Consumer  Reports  of  Financial  Assets  and  Debts 


Table  11.  Distribution  of  Use-of-Records  Ratings,  by  Respondent 
Characteristics,  Study  P3,  Wave  1 


Category 

Base 
number 

Percentage  of  category 
rated 

Characteristic 

Always 

Some- 
times 

Never 

Condition  of  home .... 

Excellent 
Good  to  excellent 
Fair  to  good 
Poor  to  fair 

42 

173 

96 

46 

31 
16 
18 
15 

48 
56 
55 
41 

21 
28 

27 
44 

Under  35 
35  to  49 
50  to  64 
65  or  over 

49 
139 
122 

52 

33 
15 
20 
10 

47 
60 
48 
50 

20 
25 
33 
40 

Size  of  savings  unit.  .  .  . 

1 

2 
3 
4 
5  or  more 

25 
113 
64 
59 
99 

20 
15 
20 
20 
18 

60 
48 
50 
47 
61 

20 
37 
30 
32 
21 

United  States 
Other 

329 

13 

18 
8 

52 
77 

30 

15 

Owner 
Tenant 

239 

123 

16 
22 

51 
56 

33 

22 

Grade  school 
High  school 
College  or  more 

126 

202 

34 

14 
19 
26 

49 
55 

47 

37 

25 
26 

Income  of  savings  unit 

Under  $2,500 
$2,500  to  $4,999 
$5,000  to  $9,999 
$10,000  to  $14,999 
$15,000  or  more 

132 

111 

80 

17 

11 

24 
16 
14 
18 
9 

48 
55 
64 
65 
27 

28 
29 
22 
18 
64 

Value  of  assets 

Under  $25,000 
$25,000  to  $49,999 
$50,000  to  $99,999 
$100,000  or  more 

68 

89 

108 

75 

16 
20 
22 
13 

59 
49 
53 
57 

25 
30 
25 
29 

Under  100  acres 
100  to  199  acres 
200  to  299  acres 
300  acres  or  more 

52 

87 

85 

137 

6 
17 
21 
22 

54 
51 
58 
50 

40 

32 
21 
28 

Amount  of  farm  debt . 

Under  $1,000 
$1,000  to  $4,999 
$5,000  to  $9,999 
$10,000  or  more 

75 

99 

73 

110 

20 
15 
29 
14 

49 
55 
49 
56 

31 
30 
22 
30 

Amount  in  savings 

None 

Under  $1,000 
$1,000  to  $4,999 
$5,000  or  more 

225 
70 
34 
20 

15 
26 
24 
25 

56 
57 
38 
40 

30 
17 
38 
35 

(( 
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